From 709d26d8805bcd08e7957758c2d685994e62aa5f Mon Sep 17 00:00:00 2001 From: Vishal Jaishankar Date: Fri, 17 Jul 2026 07:13:09 +0530 Subject: [PATCH] fix skills to discover CrdbCluster schema Update the CockroachDB Helm troubleshooting skills to discover live CrdbCluster resources before reading object-specific fields. This removes assumptions that Helm release names, service names, image tags, or CRDB versions match CrdbCluster.metadata.name. Fetch and save the served CRD schema, derive available field paths from that schema, and use those paths when collecting image, action, condition, migration, certificate, ready-node, and reconciled data. This keeps skill output aligned with the CRD version actually installed in the target cluster. Apply the same discovery-first pattern across install verification, migration debugging, TLS validation, escalation packet collection, Helm diagnosis, and multi-region service checks. Fix the schema helper for zsh by avoiding the special shell variable name path, which otherwise mutates PATH and breaks jq lookup inside crdb_first_schema_path. Handle namespaces with no live CrdbCluster rows by stopping object-specific drilldown. Remaining CrdbNode owner references and labels may be collected as teardown evidence, but they are not treated as replacement CrdbCluster discovery. Add a Codex directive requiring future PRs to include non-empty commit and PR message details covering summary, rationale, and validation. Validation: git diff --check; Markdown code fence balance check; no remaining unsafe for-path helper loops; live OpenShift cluster read with KUBECONFIG from /var/folders/8q/494h7chx3jn5291_k74_2q5c0000gp/T/helm-charts-ocp-ocp-4a34ff7a86-463512545/auth/kubeconfig confirmed empty CrdbCluster discovery in cockroach-ns-xxnxyv and resolved live CRD schema paths for spec.template.spec.image, status.actions, and spec.template.spec.certificates. --- AGENTS.md | 3 + .../SKILL.md | 237 ++++++++++---- .../SKILL.md | 291 +++++++++++++----- .../SKILL.md | 145 +++++++-- .../installing-cockroachdb-with-helm/SKILL.md | 99 +++++- .../SKILL.md | 13 +- .../configuring-cockroachdb-helm-tls/SKILL.md | 77 ++++- 7 files changed, 685 insertions(+), 180 deletions(-) create mode 100644 AGENTS.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..bb1496e3 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,3 @@ +# Codex Directives + +- When creating a pull request, do not leave the commit message or PR description empty. Use a concise subject and include enough detail in the body for reviewers to understand the summary, rationale, and validation performed. diff --git a/skills/cockroachdb-observability-and-diagnostics/collecting-cockroachdb-operator-escalation-packet/SKILL.md b/skills/cockroachdb-observability-and-diagnostics/collecting-cockroachdb-operator-escalation-packet/SKILL.md index fd4d286d..24b5a38e 100644 --- a/skills/cockroachdb-observability-and-diagnostics/collecting-cockroachdb-operator-escalation-packet/SKILL.md +++ b/skills/cockroachdb-observability-and-diagnostics/collecting-cockroachdb-operator-escalation-packet/SKILL.md @@ -31,14 +31,17 @@ Collects the artifacts needed for TSC/TSE or operator-team escalation. Use this ## Execution Discipline - Execute one step at a time and inspect the output before moving on. Do not run the packet collection in parallel; earlier results determine which operation-specific sections are relevant. +- Run every step in the same shell session. Packet collection depends on `OPERATOR_NAMESPACE`, `CRDB_NAMESPACE`, `CRDBCLUSTER`, and the `CRDB_*_PATH` exports set in Step 1; opening a new shell drops those variables and downstream `jq`/`kubectl` calls will read from the wrong object or emit empty artifacts. - Keep collection read-only unless the user explicitly approves a mutating action for the target cluster. +- Never infer a `CrdbCluster` object name from a Helm release, service name, or CockroachDB image/version string. List `CrdbCluster` objects and use the exact `metadata.name`. +- Before reading version-sensitive `CrdbCluster` fields, save the live CRD YAML and derive field paths from the served CRD schema for the object's `apiVersion`. - Do not run interactive `kubectl exec` shells, `kubectl debug`, port-forwards, pprof/metrics collection, or commands that use external images in production unless the user approves them. If impact or policy is unclear, involve TSE or the operator team first. - Do not patch, annotate, delete, restart, scale, drain, decommission, or run `helm upgrade` as part of packet collection. ## Required Inputs - Operator namespace and Helm release -- CockroachDB namespace and Helm release +- CockroachDB namespace, Helm release, and discovered `CrdbCluster.metadata.name` - Kubernetes context and cluster/provider - Current operation: install, upgrade, scale up/down, cert rotation, migration, routine maintenance, or recovery - Current and target CockroachDB image, if an upgrade is involved @@ -57,65 +60,169 @@ Use clear filenames, for example `operator-logs.txt`, `crdbcluster.yaml`, `event ## Step 1: Context and Version Inventory ```bash +export OPERATOR_NAMESPACE="" +export OPERATOR_RELEASE="" +export CRDB_NAMESPACE="" +export CRDB_HELM_RELEASE="" + kubectl config current-context > crdb-operator-escalation/kube-context.txt kubectl version --short > crdb-operator-escalation/kubernetes-version.txt -helm -n status > crdb-operator-escalation/operator-helm-status.txt 2>&1 || true -helm -n history > crdb-operator-escalation/operator-helm-history.txt 2>&1 || true -helm -n status > crdb-operator-escalation/crdb-helm-status.txt 2>&1 || true -helm -n history > crdb-operator-escalation/crdb-helm-history.txt 2>&1 || true +helm -n "$OPERATOR_NAMESPACE" status "$OPERATOR_RELEASE" > crdb-operator-escalation/operator-helm-status.txt 2>&1 || true +helm -n "$OPERATOR_NAMESPACE" history "$OPERATOR_RELEASE" > crdb-operator-escalation/operator-helm-history.txt 2>&1 || true +helm -n "$CRDB_NAMESPACE" status "$CRDB_HELM_RELEASE" > crdb-operator-escalation/crdb-helm-status.txt 2>&1 || true +helm -n "$CRDB_NAMESPACE" history "$CRDB_HELM_RELEASE" > crdb-operator-escalation/crdb-helm-history.txt 2>&1 || true -kubectl -n get deploy cockroach-operator -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' > crdb-operator-escalation/operator-image.txt +kubectl -n "$OPERATOR_NAMESPACE" get deploy cockroach-operator -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' > crdb-operator-escalation/operator-image.txt kubectl get crd crdbclusters.crdb.cockroachlabs.com crdbnodes.crdb.cockroachlabs.com -o wide > crdb-operator-escalation/crds.txt kubectl get crd crdbclusters.crdb.cockroachlabs.com -o yaml > crdb-operator-escalation/crdbclusters-crd.yaml +kubectl get crd crdbclusters.crdb.cockroachlabs.com -o json > crdb-operator-escalation/crdbclusters-crd.json kubectl get crd crdbnodes.crdb.cockroachlabs.com -o yaml > crdb-operator-escalation/crdbnodes-crd.yaml + +kubectl -n "$CRDB_NAMESPACE" get crdbcluster -o json | jq -r ' + .items[] + | [.metadata.name, .apiVersion, (.metadata.labels["app.kubernetes.io/instance"] // ""), (.metadata.generation | tostring)] + | @tsv +' > crdb-operator-escalation/crdbcluster-candidates.tsv +``` + +Inspect `crdbcluster-candidates.tsv`. If it is empty, stop object-specific packet collection and report that no live `CrdbCluster` exists in the namespace. You may collect `CrdbNode` owner references and labels as teardown evidence, but do not treat those values as a replacement for a discovered `CrdbCluster`. If more than one `CrdbCluster` exists in the namespace, select the target by `metadata.name`; do not use a CockroachDB version or image tag as the object name. + +```bash +export CRDBCLUSTER="" +test -n "$CRDBCLUSTER" + +kubectl -n "$CRDB_NAMESPACE" get crdbcluster "$CRDBCLUSTER" -o yaml > crdb-operator-escalation/crdbcluster.yaml +kubectl -n "$CRDB_NAMESPACE" get crdbcluster "$CRDBCLUSTER" -o json > crdb-operator-escalation/crdbcluster.json + +export CRDBCLUSTER_API_VERSION="$(jq -r '.apiVersion | split("/")[-1]' crdb-operator-escalation/crdbcluster.json)" +export CRDBCLUSTER_SCHEMA_JSON=crdb-operator-escalation/crdbcluster-schema.json +jq -e --arg version "$CRDBCLUSTER_API_VERSION" ' + .spec.versions[] | select(.name == $version) | .schema.openAPIV3Schema +' crdb-operator-escalation/crdbclusters-crd.json > "$CRDBCLUSTER_SCHEMA_JSON" + +crdb_schema_has() { + jq -e --arg path "$1" ' + def has_schema_path($schema; $parts): + if ($parts | length) == 0 then true + elif (($schema.properties? // {}) | has($parts[0])) then + has_schema_path($schema.properties[$parts[0]]; $parts[1:]) + else false + end; + has_schema_path(.; $path | split(".")) + ' "$CRDBCLUSTER_SCHEMA_JSON" >/dev/null +} + +crdb_first_schema_path() { + for schema_path in "$@"; do + if crdb_schema_has "$schema_path"; then + printf '%s\n' "$schema_path" + return 0 + fi + done + printf '\n' +} + +export CRDB_MODE_PATH="$(crdb_first_schema_path spec.mode)" +export CRDB_REGIONS_PATH="$(crdb_first_schema_path spec.regions)" +export CRDB_DESIRED_IMAGE_PATH="$(crdb_first_schema_path spec.template.spec.image spec.image.name spec.image)" +export CRDB_OBSERVED_GENERATION_PATH="$(crdb_first_schema_path status.observedGeneration)" +export CRDB_RECONCILED_PATH="$(crdb_first_schema_path status.reconciled)" +export CRDB_READY_NODES_PATH="$(crdb_first_schema_path status.readyNodes)" +export CRDB_STATUS_IMAGE_PATH="$(crdb_first_schema_path status.image status.crdbcontainerimage)" +export CRDB_STATUS_VERSION_PATH="$(crdb_first_schema_path status.version)" +export CRDB_ACTIONS_PATH="$(crdb_first_schema_path status.actions status.operatorActions)" +export CRDB_CONDITIONS_PATH="$(crdb_first_schema_path status.conditions)" + +printf '%s\n' \ + "apiVersion=$CRDBCLUSTER_API_VERSION" \ + "mode=$CRDB_MODE_PATH" \ + "regions=$CRDB_REGIONS_PATH" \ + "desiredImage=$CRDB_DESIRED_IMAGE_PATH" \ + "observedGeneration=$CRDB_OBSERVED_GENERATION_PATH" \ + "reconciled=$CRDB_RECONCILED_PATH" \ + "readyNodes=$CRDB_READY_NODES_PATH" \ + "statusImage=$CRDB_STATUS_IMAGE_PATH" \ + "statusVersion=$CRDB_STATUS_VERSION_PATH" \ + "actions=$CRDB_ACTIONS_PATH" \ + "conditions=$CRDB_CONDITIONS_PATH" \ + > crdb-operator-escalation/crdbcluster-schema-paths.txt ``` Get the CockroachDB version from a Ready pod: ```bash -kubectl -n exec -c cockroachdb -- \ +kubectl -n "$CRDB_NAMESPACE" exec -c cockroachdb -- \ /cockroach/cockroach version > crdb-operator-escalation/crdb-version.txt 2>&1 ``` ## Step 2: Resource Specifications and Status ```bash -kubectl -n get deploy,pod,svc,endpoints -o wide > crdb-operator-escalation/operator-resources.txt -kubectl -n describe deploy cockroach-operator > crdb-operator-escalation/operator-deploy-describe.txt -kubectl -n describe pods -l app=cockroach-operator > crdb-operator-escalation/operator-pods-describe.txt -kubectl -n get deploy cockroach-operator -o yaml > crdb-operator-escalation/operator-deploy.yaml - -kubectl -n get crdbcluster -o yaml > crdb-operator-escalation/crdbcluster.yaml -kubectl -n describe crdbcluster > crdb-operator-escalation/crdbcluster-describe.txt -kubectl -n get crdbnodes -o yaml > crdb-operator-escalation/crdbnodes.yaml -kubectl -n describe crdbnodes > crdb-operator-escalation/crdbnodes-describe.txt -kubectl -n get pod,svc,endpoints,pvc,pdb -o wide > crdb-operator-escalation/crdb-resources-wide.txt -kubectl -n describe pods -l app.kubernetes.io/name=cockroachdb > crdb-operator-escalation/crdb-pods-describe.txt -kubectl -n describe pvc > crdb-operator-escalation/pvc-describe.txt -kubectl -n describe pdb > crdb-operator-escalation/pdb-describe.txt -kubectl -n get events --sort-by=.lastTimestamp > crdb-operator-escalation/events.txt +kubectl -n "$OPERATOR_NAMESPACE" get deploy,pod,svc,endpoints -o wide > crdb-operator-escalation/operator-resources.txt +kubectl -n "$OPERATOR_NAMESPACE" describe deploy cockroach-operator > crdb-operator-escalation/operator-deploy-describe.txt +kubectl -n "$OPERATOR_NAMESPACE" describe pods -l app=cockroach-operator > crdb-operator-escalation/operator-pods-describe.txt +kubectl -n "$OPERATOR_NAMESPACE" get deploy cockroach-operator -o yaml > crdb-operator-escalation/operator-deploy.yaml + +kubectl -n "$CRDB_NAMESPACE" describe crdbcluster "$CRDBCLUSTER" > crdb-operator-escalation/crdbcluster-describe.txt +kubectl -n "$CRDB_NAMESPACE" get crdbnodes -o yaml > crdb-operator-escalation/crdbnodes.yaml +kubectl -n "$CRDB_NAMESPACE" describe crdbnodes > crdb-operator-escalation/crdbnodes-describe.txt +kubectl -n "$CRDB_NAMESPACE" get pod,svc,endpoints,pvc,pdb -o wide > crdb-operator-escalation/crdb-resources-wide.txt +kubectl -n "$CRDB_NAMESPACE" describe pods -l app.kubernetes.io/name=cockroachdb > crdb-operator-escalation/crdb-pods-describe.txt +kubectl -n "$CRDB_NAMESPACE" describe pvc > crdb-operator-escalation/pvc-describe.txt +kubectl -n "$CRDB_NAMESPACE" describe pdb > crdb-operator-escalation/pdb-describe.txt +kubectl -n "$CRDB_NAMESPACE" get events --sort-by=.lastTimestamp > crdb-operator-escalation/events.txt ``` Summarize key cluster status: ```bash -kubectl -n get crdbcluster -o json | jq '{ - mode: .spec.mode, - nodes: [.spec.regions[]?.nodes], - regions: .spec.regions, - image: .spec.image, +jq \ + --arg modePath "$CRDB_MODE_PATH" \ + --arg regionsPath "$CRDB_REGIONS_PATH" \ + --arg desiredImagePath "$CRDB_DESIRED_IMAGE_PATH" \ + --arg observedGenerationPath "$CRDB_OBSERVED_GENERATION_PATH" \ + --arg reconciledPath "$CRDB_RECONCILED_PATH" \ + --arg readyNodesPath "$CRDB_READY_NODES_PATH" \ + --arg statusImagePath "$CRDB_STATUS_IMAGE_PATH" \ + --arg statusVersionPath "$CRDB_STATUS_VERSION_PATH" \ + --arg actionsPath "$CRDB_ACTIONS_PATH" \ + --arg conditionsPath "$CRDB_CONDITIONS_PATH" \ + ' + def value($path): if $path == "" then null else getpath($path | split(".")) end; + { + apiVersion, + name: .metadata.name, + schemaPaths: { + mode: $modePath, + regions: $regionsPath, + desiredImage: $desiredImagePath, + observedGeneration: $observedGenerationPath, + reconciled: $reconciledPath, + readyNodes: $readyNodesPath, + statusImage: $statusImagePath, + statusVersion: $statusVersionPath, + actions: $actionsPath, + conditions: $conditionsPath + }, + mode: value($modePath), + nodes: (value($regionsPath) // [] | map(.nodes)), + regions: value($regionsPath), + desiredImage: value($desiredImagePath), generation: .metadata.generation, - observedGeneration: .status.observedGeneration, - statusImage: .status.image, - actions: .status.actions, - conditions: .status.conditions, + observedGeneration: value($observedGenerationPath), + reconciled: value($reconciledPath), + readyNodes: value($readyNodesPath), + statusImage: value($statusImagePath), + statusVersion: value($statusVersionPath), + actions: value($actionsPath), + conditions: value($conditionsPath), labels: .metadata.labels, annotations: .metadata.annotations -}' > crdb-operator-escalation/crdbcluster-summary.json +}' crdb-operator-escalation/crdbcluster.json > crdb-operator-escalation/crdbcluster-summary.json -kubectl -n get crdbnodes \ - -o custom-columns=NAME:.metadata.name,GENERATION:.metadata.generation,OBSERVED:.status.observedGeneration,PHASE:.status.phase,HASH:.metadata.annotations["crdb.cockroachlabs.com/hash-revision"],NODE_ID:.status.nodeID \ +kubectl -n "$CRDB_NAMESPACE" get crdbnodes \ + -o 'custom-columns=NAME:.metadata.name,GENERATION:.metadata.generation,OBSERVED:.status.observedGeneration,DECOMMISSION:.status.decommission,REVISION:.metadata.annotations.crdb\.cockroachlabs\.com/clusterNodeRevision,NODE_ID:.status.nodeID' \ > crdb-operator-escalation/crdbnodes-summary.txt ``` @@ -124,16 +231,16 @@ kubectl -n get crdbnodes \ Collect full recent logs, not filtered snippets: ```bash -kubectl -n logs -l app=cockroach-operator --tail=500 > crdb-operator-escalation/operator-logs.txt 2>&1 -kubectl -n logs -l app=cockroach-operator --previous --tail=500 > crdb-operator-escalation/operator-previous-logs.txt 2>&1 || true +kubectl -n "$OPERATOR_NAMESPACE" logs -l app=cockroach-operator --tail=500 > crdb-operator-escalation/operator-logs.txt 2>&1 +kubectl -n "$OPERATOR_NAMESPACE" logs -l app=cockroach-operator --previous --tail=500 > crdb-operator-escalation/operator-previous-logs.txt 2>&1 || true ``` For each CockroachDB pod: ```bash -kubectl -n logs -c cockroachdb --tail=500 > crdb-operator-escalation/-cockroachdb.log 2>&1 -kubectl -n logs -c cockroachdb --previous --tail=500 > crdb-operator-escalation/-cockroachdb-previous.log 2>&1 || true -kubectl -n logs -c cert-reloader --tail=100 > crdb-operator-escalation/-cert-reloader.log 2>&1 || true +kubectl -n "$CRDB_NAMESPACE" logs -c cockroachdb --tail=500 > crdb-operator-escalation/-cockroachdb.log 2>&1 +kubectl -n "$CRDB_NAMESPACE" logs -c cockroachdb --previous --tail=500 > crdb-operator-escalation/-cockroachdb-previous.log 2>&1 || true +kubectl -n "$CRDB_NAMESPACE" logs -c cert-reloader --tail=100 > crdb-operator-escalation/-cert-reloader.log 2>&1 || true ``` ## Step 4: Operation-Specific Evidence @@ -141,18 +248,34 @@ kubectl -n logs -c cert-reloader --tail=100 > ### Upgrade ```bash -kubectl -n get crdbcluster -o json | jq '{ - specImage: .spec.image, - statusImage: .status.image, - actions: .status.actions, +jq \ + --arg desiredImagePath "$CRDB_DESIRED_IMAGE_PATH" \ + --arg statusImagePath "$CRDB_STATUS_IMAGE_PATH" \ + --arg statusVersionPath "$CRDB_STATUS_VERSION_PATH" \ + --arg actionsPath "$CRDB_ACTIONS_PATH" \ + --arg conditionsPath "$CRDB_CONDITIONS_PATH" \ + ' + def value($path): if $path == "" then null else getpath($path | split(".")) end; + { + apiVersion, + name: .metadata.name, + desiredImagePath: $desiredImagePath, + desiredImage: value($desiredImagePath), + statusImagePath: $statusImagePath, + statusImage: value($statusImagePath), + statusVersionPath: $statusVersionPath, + statusVersion: value($statusVersionPath), + actionsPath: $actionsPath, + actions: value($actionsPath), + conditionsPath: $conditionsPath, + conditions: value($conditionsPath), annotations: .metadata.annotations, - conditions: [.status.conditions[]? | select(.type | test("Upgrade|Version|Validate"))] -}' > crdb-operator-escalation/upgrade-status.json +}' crdb-operator-escalation/crdbcluster.json > crdb-operator-escalation/upgrade-status.json -kubectl -n get jobs -o wide > crdb-operator-escalation/jobs.txt -kubectl -n get pods -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image,PHASE:.status.phase > crdb-operator-escalation/pod-images.txt -kubectl -n describe job > crdb-operator-escalation/version-checker-job.txt 2>&1 || true -kubectl -n logs -l job-name= > crdb-operator-escalation/version-checker-logs.txt 2>&1 || true +kubectl -n "$CRDB_NAMESPACE" get jobs -o wide > crdb-operator-escalation/jobs.txt +kubectl -n "$CRDB_NAMESPACE" get pods -o 'custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image,PHASE:.status.phase' > crdb-operator-escalation/pod-images.txt +kubectl -n "$CRDB_NAMESPACE" describe job > crdb-operator-escalation/version-checker-job.txt 2>&1 || true +kubectl -n "$CRDB_NAMESPACE" logs -l job-name= > crdb-operator-escalation/version-checker-logs.txt 2>&1 || true ``` Include current and target CRDB image and whether any version checker job or pod was deleted. @@ -160,10 +283,10 @@ Include current and target CRDB image and whether any version checker job or pod ### Scale Down or Decommission ```bash -kubectl -n exec -c cockroachdb -- \ +kubectl -n "$CRDB_NAMESPACE" exec -c cockroachdb -- \ /cockroach/cockroach node status --decommission > crdb-operator-escalation/node-decommission-status.txt 2>&1 -kubectl -n get crdbnodes -o json | jq '[.items[] | select(.status.phase=="Decommissioning")] | {count: length, nodes: [.[].metadata.name]}' \ +kubectl -n "$CRDB_NAMESPACE" get crdbnodes -o json | jq '[.items[] | select(.status.decommission != null and .status.decommission != "")] | {count: length, nodes: [.[] | {name: .metadata.name, decommission: .status.decommission}]}' \ > crdb-operator-escalation/decommissioning-crdbnodes.json ``` @@ -176,16 +299,16 @@ Capture whether machines, Kubernetes nodes, disks, storage classes, topology spr ```bash kubectl get nodes -L topology.kubernetes.io/region,topology.kubernetes.io/zone > crdb-operator-escalation/nodes-locality.txt kubectl get storageclass > crdb-operator-escalation/storageclasses.txt -kubectl -n get pvc -o wide > crdb-operator-escalation/pvc-wide.txt +kubectl -n "$CRDB_NAMESPACE" get pvc -o wide > crdb-operator-escalation/pvc-wide.txt ``` ### Certificate Rotation ```bash -kubectl -n get secret,configmap | grep -E 'ca|node|client|tls|cert' > crdb-operator-escalation/cert-resources.txt || true -kubectl -n get certificate,issuer,clusterissuer -o wide > crdb-operator-escalation/cert-manager-resources.txt 2>&1 || true +kubectl -n "$CRDB_NAMESPACE" get secret,configmap | grep -E 'ca|node|client|tls|cert' > crdb-operator-escalation/cert-resources.txt || true +kubectl -n "$CRDB_NAMESPACE" get certificate,issuer,clusterissuer -o wide > crdb-operator-escalation/cert-manager-resources.txt 2>&1 || true -kubectl -n get secret -o jsonpath='{.data.tls\.crt}' | base64 -d | \ +kubectl -n "$CRDB_NAMESPACE" get secret -o jsonpath='{.data.tls\.crt}' | base64 -d | \ openssl x509 -noout -dates -subject -issuer -ext subjectAltName > crdb-operator-escalation/node-cert-metadata.txt ``` @@ -202,7 +325,7 @@ Collect these when the operator is running but not reconciling, logs/status do n In one terminal: ```bash -kubectl -n port-forward deployment/cockroach-operator 7080:7080 +kubectl -n "$OPERATOR_NAMESPACE" port-forward deployment/cockroach-operator 7080:7080 ``` In another terminal: @@ -218,7 +341,7 @@ curl -s 'http://localhost:7080/debug/pprof/mutex' > crdb-operator-escalation/mut Metrics: ```bash -kubectl -n port-forward deployment/cockroach-operator 8080:8080 +kubectl -n "$OPERATOR_NAMESPACE" port-forward deployment/cockroach-operator 8080:8080 curl -s http://localhost:8080/metrics > crdb-operator-escalation/metrics_dump.txt grep 'controller_runtime_reconcile_total' crdb-operator-escalation/metrics_dump.txt > crdb-operator-escalation/reconcile-total.txt || true grep 'controller_runtime_reconcile_errors_total' crdb-operator-escalation/metrics_dump.txt > crdb-operator-escalation/reconcile-errors.txt || true @@ -253,7 +376,7 @@ Return a concise summary with: 1. Operation type and current impact 2. Current operator and CockroachDB versions -3. Current `CrdbCluster` mode, generation, observedGeneration, status image, and actions +3. Current schema-grounded `CrdbCluster` mode, generation, observedGeneration, image, version, actions, and conditions 4. Stuck resource or symptom 5. Any unsafe operations already performed 6. Missing artifacts, if any diff --git a/skills/cockroachdb-observability-and-diagnostics/diagnosing-cockroachdb-helm-deployments/SKILL.md b/skills/cockroachdb-observability-and-diagnostics/diagnosing-cockroachdb-helm-deployments/SKILL.md index d428968c..2682177e 100644 --- a/skills/cockroachdb-observability-and-diagnostics/diagnosing-cockroachdb-helm-deployments/SKILL.md +++ b/skills/cockroachdb-observability-and-diagnostics/diagnosing-cockroachdb-helm-deployments/SKILL.md @@ -52,37 +52,120 @@ Diagnoses CockroachDB Helm install, upgrade, and readiness failures for operator ## Execution Discipline - Execute one step at a time and inspect the output before moving on. Do not run whole sections, unrelated command groups, or later diagnostic branches in parallel; earlier output determines which later checks are relevant. +- Run every step in the same shell session. The commands rely on `OPERATOR_NAMESPACE`, `CRDB_NAMESPACE`, `CRDBCLUSTER`, `CRDB_DIAG_DIR`, and the `CRDB_*_PATH` exports set in Step 1; opening a new shell drops those variables and later steps will hit unresolved names or read the wrong object. - Treat commands as templates. Substitute namespaces, release names, chart names, and pod names deliberately before running anything. +- Never infer a `CrdbCluster` object name from a Helm release, service name, or CockroachDB image/version string. List `CrdbCluster` objects and use the exact `metadata.name`. +- Before reading version-sensitive `CrdbCluster` fields, save the live CRD YAML and derive field paths from the served CRD schema for the object's `apiVersion`. - Do not run any mutating command unless the user explicitly approves it for the target cluster. This includes `kubectl patch`, `kubectl annotate`, `kubectl delete`, `kubectl scale`, `kubectl rollout restart`, `helm upgrade`, drain/decommission commands, and interactive `kubectl exec` or `kubectl debug` shells. - In production or whenever the impact is unclear, stop and escalate to TSE or the operator team before pprof/metrics collection, debug containers, timestamp-based rolling restarts, mode changes, operator restarts, scale changes, or decommission actions. ## Step 1: Collect Baseline State ```bash +export OPERATOR_NAMESPACE="" +export OPERATOR_RELEASE="" +export CRDB_NAMESPACE="" +export CRDB_HELM_RELEASE="" +export CRDB_DIAG_DIR="${CRDB_DIAG_DIR:-$(mktemp -d)}" + # Helm release state -helm -n status || true -helm -n status || true -helm -n history || true -helm -n history || true +helm -n "$OPERATOR_NAMESPACE" status "$OPERATOR_RELEASE" || true +helm -n "$CRDB_NAMESPACE" status "$CRDB_HELM_RELEASE" || true +helm -n "$OPERATOR_NAMESPACE" history "$OPERATOR_RELEASE" || true +helm -n "$CRDB_NAMESPACE" history "$CRDB_HELM_RELEASE" || true # Operator state -kubectl -n get deploy,pod,svc -o wide | grep -E 'cockroach-operator|NAME' -kubectl -n logs -l app=cockroach-operator --tail=200 || true +kubectl -n "$OPERATOR_NAMESPACE" get deploy,pod,svc -o wide | grep -E 'cockroach-operator|NAME' +kubectl -n "$OPERATOR_NAMESPACE" logs -l app=cockroach-operator --tail=200 || true # CRD and CockroachDB resources kubectl get crd crdbclusters.crdb.cockroachlabs.com crdbnodes.crdb.cockroachlabs.com -kubectl -n get crdbcluster,crdbnode,pod,svc,endpoints,pvc,pdb -o wide -kubectl -n describe crdbcluster || true -kubectl -n get events --sort-by=.lastTimestamp | tail -50 +kubectl get crd crdbclusters.crdb.cockroachlabs.com -o yaml > "$CRDB_DIAG_DIR/crdbclusters-crd.yaml" +kubectl get crd crdbclusters.crdb.cockroachlabs.com -o json > "$CRDB_DIAG_DIR/crdbclusters-crd.json" +kubectl -n "$CRDB_NAMESPACE" get crdbcluster,crdbnode,pod,svc,endpoints,pvc,pdb -o wide + +kubectl -n "$CRDB_NAMESPACE" get crdbcluster -o json | jq -r ' + .items[] + | [.metadata.name, .apiVersion, (.metadata.labels["app.kubernetes.io/instance"] // ""), (.metadata.generation | tostring)] + | @tsv +' +``` + +If no `CrdbCluster` rows are returned, stop the object-specific diagnosis and report that no live `CrdbCluster` exists in the namespace. You may collect `CrdbNode` owner references and labels as teardown evidence, but do not treat those values as a replacement for a discovered `CrdbCluster`. + +If multiple `CrdbCluster` rows are returned, choose the target by `metadata.name`. Do not use the Helm release or a CockroachDB version as a substitute. + +```bash +export CRDBCLUSTER="" +test -n "$CRDBCLUSTER" + +kubectl -n "$CRDB_NAMESPACE" get crdbcluster "$CRDBCLUSTER" -o yaml > "$CRDB_DIAG_DIR/crdbcluster.yaml" +kubectl -n "$CRDB_NAMESPACE" get crdbcluster "$CRDBCLUSTER" -o json > "$CRDB_DIAG_DIR/crdbcluster.json" +kubectl -n "$CRDB_NAMESPACE" describe crdbcluster "$CRDBCLUSTER" || true +kubectl -n "$CRDB_NAMESPACE" get events --sort-by=.lastTimestamp | tail -50 + +export CRDBCLUSTER_API_VERSION="$(jq -r '.apiVersion | split("/")[-1]' "$CRDB_DIAG_DIR/crdbcluster.json")" +export CRDBCLUSTER_SCHEMA_JSON="$CRDB_DIAG_DIR/crdbcluster-schema.json" +jq -e --arg version "$CRDBCLUSTER_API_VERSION" ' + .spec.versions[] | select(.name == $version) | .schema.openAPIV3Schema +' "$CRDB_DIAG_DIR/crdbclusters-crd.json" > "$CRDBCLUSTER_SCHEMA_JSON" + +crdb_schema_has() { + jq -e --arg path "$1" ' + def has_schema_path($schema; $parts): + if ($parts | length) == 0 then true + elif (($schema.properties? // {}) | has($parts[0])) then + has_schema_path($schema.properties[$parts[0]]; $parts[1:]) + else false + end; + has_schema_path(.; $path | split(".")) + ' "$CRDBCLUSTER_SCHEMA_JSON" >/dev/null +} + +crdb_first_schema_path() { + for schema_path in "$@"; do + if crdb_schema_has "$schema_path"; then + printf '%s\n' "$schema_path" + return 0 + fi + done + printf '\n' +} + +export CRDB_MODE_PATH="$(crdb_first_schema_path spec.mode)" +export CRDB_REGIONS_PATH="$(crdb_first_schema_path spec.regions)" +export CRDB_DESIRED_IMAGE_PATH="$(crdb_first_schema_path spec.template.spec.image spec.image.name spec.image)" +export CRDB_OBSERVED_GENERATION_PATH="$(crdb_first_schema_path status.observedGeneration)" +export CRDB_RECONCILED_PATH="$(crdb_first_schema_path status.reconciled)" +export CRDB_READY_NODES_PATH="$(crdb_first_schema_path status.readyNodes)" +export CRDB_STATUS_IMAGE_PATH="$(crdb_first_schema_path status.image status.crdbcontainerimage)" +export CRDB_STATUS_VERSION_PATH="$(crdb_first_schema_path status.version)" +export CRDB_ACTIONS_PATH="$(crdb_first_schema_path status.actions status.operatorActions)" +export CRDB_CONDITIONS_PATH="$(crdb_first_schema_path status.conditions)" +export CRDB_CERTIFICATES_PATH="$(crdb_first_schema_path spec.template.spec.certificates spec.certificates)" + +printf '%s\n' \ + "apiVersion=$CRDBCLUSTER_API_VERSION" \ + "mode=$CRDB_MODE_PATH" \ + "regions=$CRDB_REGIONS_PATH" \ + "desiredImage=$CRDB_DESIRED_IMAGE_PATH" \ + "observedGeneration=$CRDB_OBSERVED_GENERATION_PATH" \ + "reconciled=$CRDB_RECONCILED_PATH" \ + "readyNodes=$CRDB_READY_NODES_PATH" \ + "statusImage=$CRDB_STATUS_IMAGE_PATH" \ + "statusVersion=$CRDB_STATUS_VERSION_PATH" \ + "actions=$CRDB_ACTIONS_PATH" \ + "conditions=$CRDB_CONDITIONS_PATH" \ + "certificates=$CRDB_CERTIFICATES_PATH" ``` For a stuck pod or node: ```bash -kubectl -n describe pod -kubectl -n logs -c cockroachdb --tail=200 -kubectl -n logs -c cockroachdb --previous -kubectl -n describe crdbnode +kubectl -n "$CRDB_NAMESPACE" describe pod +kubectl -n "$CRDB_NAMESPACE" logs -c cockroachdb --tail=200 +kubectl -n "$CRDB_NAMESPACE" logs -c cockroachdb --previous +kubectl -n "$CRDB_NAMESPACE" describe crdbnode ``` ## Step 2: Classify the Failure @@ -105,10 +188,10 @@ kubectl -n describe crdbnode ## CRD and Operator Readiness ```bash -kubectl -n rollout status deploy/cockroach-operator --timeout=5m +kubectl -n "$OPERATOR_NAMESPACE" rollout status deploy/cockroach-operator --timeout=5m kubectl get crd crdbclusters.crdb.cockroachlabs.com -o jsonpath='{.spec.versions[*].name}{"\n"}' kubectl get crd crdbnodes.crdb.cockroachlabs.com -o jsonpath='{.spec.versions[*].name}{"\n"}' -kubectl -n get deploy cockroach-operator -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' +kubectl -n "$OPERATOR_NAMESPACE" get deploy cockroach-operator -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' ``` Remediation: @@ -121,10 +204,10 @@ Remediation: ## Operator Health ```bash -kubectl -n get pods -l app=cockroach-operator -o wide -kubectl -n describe pod -kubectl -n logs -l app=cockroach-operator --tail=100 -kubectl -n get deploy cockroach-operator -o jsonpath='{.spec.template.spec.containers[0].env}{"\n"}' +kubectl -n "$OPERATOR_NAMESPACE" get pods -l app=cockroach-operator -o wide +kubectl -n "$OPERATOR_NAMESPACE" describe pod +kubectl -n "$OPERATOR_NAMESPACE" logs -l app=cockroach-operator --tail=100 +kubectl -n "$OPERATOR_NAMESPACE" get deploy cockroach-operator -o jsonpath='{.spec.template.spec.containers[0].env}{"\n"}' ``` Interpretation: @@ -138,13 +221,13 @@ Interpretation: Check recent logs to see whether reconciliation is active: ```bash -kubectl -n logs -l app=cockroach-operator --tail=100 | grep -i reconcil || true +kubectl -n "$OPERATOR_NAMESPACE" logs -l app=cockroach-operator --tail=100 | grep -i reconcil || true ``` Do not add ad hoc annotations to trigger reconciliation. If a user-approved reconcile-triggering change is required, use the chart-supported timestamp path through `helm upgrade --reuse-values`; this updates `helm.sh/restartedAt` and may roll CockroachDB pods, so treat it as a mutating operation: ```bash -helm -n upgrade \ +helm -n "$CRDB_NAMESPACE" upgrade "$CRDB_HELM_RELEASE" \ --reuse-values \ --set-string cockroachdb.crdbCluster.timestamp="$(date -u +%Y-%m-%dT%H:%M:%SZ)" ``` @@ -183,8 +266,8 @@ Do not set `nodeReader.create=false` before replacement RBAC exists. ## Webhook Checks ```bash -kubectl -n get svc cockroach-webhook-service -kubectl -n get endpoints cockroach-webhook-service +kubectl -n "$OPERATOR_NAMESPACE" get svc cockroach-webhook-service +kubectl -n "$OPERATOR_NAMESPACE" get endpoints cockroach-webhook-service kubectl get validatingwebhookconfigurations | grep cockroach ``` @@ -200,20 +283,50 @@ For scoped operators, webhook configurations may be namespace-suffixed, such as ## Reconciliation Not Progressing ```bash -kubectl -n get crdbcluster -o json | jq '{ - mode: .spec.mode, - image: .spec.image, +jq \ + --arg modePath "$CRDB_MODE_PATH" \ + --arg desiredImagePath "$CRDB_DESIRED_IMAGE_PATH" \ + --arg observedGenerationPath "$CRDB_OBSERVED_GENERATION_PATH" \ + --arg reconciledPath "$CRDB_RECONCILED_PATH" \ + --arg readyNodesPath "$CRDB_READY_NODES_PATH" \ + --arg statusImagePath "$CRDB_STATUS_IMAGE_PATH" \ + --arg statusVersionPath "$CRDB_STATUS_VERSION_PATH" \ + --arg actionsPath "$CRDB_ACTIONS_PATH" \ + --arg conditionsPath "$CRDB_CONDITIONS_PATH" \ + ' + def value($path): if $path == "" then null else getpath($path | split(".")) end; + { + apiVersion, + name: .metadata.name, + schemaPaths: { + mode: $modePath, + desiredImage: $desiredImagePath, + observedGeneration: $observedGenerationPath, + reconciled: $reconciledPath, + readyNodes: $readyNodesPath, + statusImage: $statusImagePath, + statusVersion: $statusVersionPath, + actions: $actionsPath, + conditions: $conditionsPath + }, + mode: value($modePath), + desiredImage: value($desiredImagePath), generation: .metadata.generation, - observedGeneration: .status.observedGeneration, - statusImage: .status.image, - actions: .status.actions, - conditions: .status.conditions -}' - -kubectl -n get crdbnodes \ - -o custom-columns=NAME:.metadata.name,GENERATION:.metadata.generation,OBSERVED:.status.observedGeneration,PHASE:.status.phase,HASH:.metadata.annotations["crdb.cockroachlabs.com/hash-revision"],NODE_ID:.status.nodeID + observedGeneration: value($observedGenerationPath), + reconciled: value($reconciledPath), + readyNodes: value($readyNodesPath), + statusImage: value($statusImagePath), + statusVersion: value($statusVersionPath), + actions: value($actionsPath), + conditions: value($conditionsPath) +}' "$CRDB_DIAG_DIR/crdbcluster.json" + +kubectl -n "$CRDB_NAMESPACE" get crdbnodes \ + -o 'custom-columns=NAME:.metadata.name,GENERATION:.metadata.generation,OBSERVED:.status.observedGeneration,DECOMMISSION:.status.decommission,REVISION:.metadata.annotations.crdb\.cockroachlabs\.com/clusterNodeRevision,NODE_ID:.status.nodeID' ``` +`CrdbNode` status has no `phase` field. Use `.status.decommission` to see the current decommission state (empty for healthy nodes; `draining`, `drained`, `transferringReplicas`, `zeroReplicas`, or `decommissioned` otherwise) and the `crdb.cockroachlabs.com/clusterNodeRevision` annotation to compare the current revision against the operator's desired revision. Wrap the whole `-o custom-columns=...` value in single quotes so shells (especially zsh) do not glob the `[]` or consume the internal separators. + Checklist: 1. Confirm the operator is running and watching the CockroachDB namespace. @@ -225,13 +338,13 @@ Checklist: ## Pod Readiness and CRDB Issues ```bash -kubectl -n get pods -l app.kubernetes.io/name=cockroachdb -o wide -kubectl -n describe pod -kubectl -n logs -c cockroachdb --tail=200 -kubectl -n logs -c cockroachdb --previous -kubectl -n get pod -o jsonpath='{.spec.containers[0].readinessProbe}{"\n"}' -kubectl -n get pods -l app.kubernetes.io/name=cockroachdb \ - -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image,PHASE:.status.phase,READY:.status.containerStatuses[0].ready,NODE:.spec.nodeName +kubectl -n "$CRDB_NAMESPACE" get pods -l app.kubernetes.io/name=cockroachdb -o wide +kubectl -n "$CRDB_NAMESPACE" describe pod +kubectl -n "$CRDB_NAMESPACE" logs -c cockroachdb --tail=200 +kubectl -n "$CRDB_NAMESPACE" logs -c cockroachdb --previous +kubectl -n "$CRDB_NAMESPACE" get pod -o jsonpath='{.spec.containers[0].readinessProbe}{"\n"}' +kubectl -n "$CRDB_NAMESPACE" get pods -l app.kubernetes.io/name=cockroachdb \ + -o 'custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image,PHASE:.status.phase,READY:.status.containerStatuses[0].ready,NODE:.spec.nodeName' ``` Common pod issues: @@ -243,24 +356,41 @@ Common pod issues: ## Upgrade and Version Validation ```bash -kubectl -n get crdbcluster -o json | jq '{ - specImage: .spec.image, - statusImage: .status.image, - actions: .status.actions, - conditions: [.status.conditions[]? | select(.type | test("Upgrade|Version|Validate"))] -}' - -kubectl -n get crdbcluster -o jsonpath='{.metadata.annotations}{"\n"}' | jq . -kubectl -n get jobs -kubectl -n describe job -kubectl -n logs -l job-name= -kubectl -n get pods -l app.kubernetes.io/name=cockroachdb \ - -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image,REVISION:.metadata.annotations["crdb\.cockroachlabs\.com/hash-revision"],PHASE:.status.phase +jq \ + --arg desiredImagePath "$CRDB_DESIRED_IMAGE_PATH" \ + --arg statusImagePath "$CRDB_STATUS_IMAGE_PATH" \ + --arg statusVersionPath "$CRDB_STATUS_VERSION_PATH" \ + --arg actionsPath "$CRDB_ACTIONS_PATH" \ + --arg conditionsPath "$CRDB_CONDITIONS_PATH" \ + ' + def value($path): if $path == "" then null else getpath($path | split(".")) end; + { + apiVersion, + name: .metadata.name, + desiredImagePath: $desiredImagePath, + desiredImage: value($desiredImagePath), + statusImagePath: $statusImagePath, + statusImage: value($statusImagePath), + statusVersionPath: $statusVersionPath, + statusVersion: value($statusVersionPath), + actionsPath: $actionsPath, + actions: value($actionsPath), + conditionsPath: $conditionsPath, + conditions: value($conditionsPath), + annotations: .metadata.annotations +}' "$CRDB_DIAG_DIR/crdbcluster.json" + +jq '.metadata.annotations' "$CRDB_DIAG_DIR/crdbcluster.json" +kubectl -n "$CRDB_NAMESPACE" get jobs +kubectl -n "$CRDB_NAMESPACE" describe job +kubectl -n "$CRDB_NAMESPACE" logs -l job-name= +kubectl -n "$CRDB_NAMESPACE" get pods -l app.kubernetes.io/name=cockroachdb \ + -o 'custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image,REVISION:.metadata.annotations.crdb\.cockroachlabs\.com/nodeRevision,PHASE:.status.phase' ``` Interpretation: -- If `spec.image` differs from `status.image`, an upgrade is in progress or stuck. +- If the schema-grounded desired image path differs from the schema-grounded status image path, an upgrade is in progress or stuck. - If a rejected-image annotation exists, inspect its value; the operator rejected the target version. - If the version checker job exists but the pod is gone, use job status and operator logs for validation messages. - Do not delete version checker jobs or pods until their status and logs are captured. @@ -270,16 +400,22 @@ Interpretation: The operator creates separate service paths for pod DNS and join traffic. Do not change service settings without operator-team guidance. ```bash -kubectl -n get service -o yaml -kubectl -n get service -join -o yaml -kubectl -n get endpoints -kubectl -n get endpoints -join - -kubectl -n exec -c cockroachdb -- \ - nslookup ..svc.cluster.local 2>&1 || true - -kubectl -n exec -c cockroachdb -- \ - nslookup -join..svc.cluster.local 2>&1 || true +kubectl -n "$CRDB_NAMESPACE" get service,endpoints -o wide +export CRDB_SERVICE="" +export CRDB_JOIN_SERVICE="" +test -n "$CRDB_SERVICE" +test -n "$CRDB_JOIN_SERVICE" + +kubectl -n "$CRDB_NAMESPACE" get service "$CRDB_SERVICE" -o yaml +kubectl -n "$CRDB_NAMESPACE" get service "$CRDB_JOIN_SERVICE" -o yaml +kubectl -n "$CRDB_NAMESPACE" get endpoints "$CRDB_SERVICE" +kubectl -n "$CRDB_NAMESPACE" get endpoints "$CRDB_JOIN_SERVICE" + +kubectl -n "$CRDB_NAMESPACE" exec -c cockroachdb -- \ + nslookup "$CRDB_SERVICE.$CRDB_NAMESPACE.svc.cluster.local" 2>&1 || true + +kubectl -n "$CRDB_NAMESPACE" exec -c cockroachdb -- \ + nslookup "$CRDB_JOIN_SERVICE.$CRDB_NAMESPACE.svc.cluster.local" 2>&1 || true ``` For multi-region checks, use [validating-cockroachdb-helm-multiregion](../../cockroachdb-onboarding-and-migrations/validating-cockroachdb-helm-multiregion/SKILL.md). @@ -292,10 +428,13 @@ Quick checks: ```bash helm template -n -f values.yaml >/tmp/rendered.yaml -kubectl -n get secret,configmap | grep -E 'cockroach|crdb|cert|ca|tls' -kubectl -n get crdbcluster -o yaml | grep -A20 certificates -kubectl -n get pod -o jsonpath='{.spec.containers[*].name}{"\n"}' -kubectl -n logs -c cert-reloader --tail=100 +kubectl -n "$CRDB_NAMESPACE" get secret,configmap | grep -E 'cockroach|crdb|cert|ca|tls' +jq --arg certificatesPath "$CRDB_CERTIFICATES_PATH" ' + def value($path): if $path == "" then null else getpath($path | split(".")) end; + {certificatesPath: $certificatesPath, certificates: value($certificatesPath)} +' "$CRDB_DIAG_DIR/crdbcluster.json" +kubectl -n "$CRDB_NAMESPACE" get pod -o jsonpath='{.spec.containers[*].name}{"\n"}' +kubectl -n "$CRDB_NAMESPACE" logs -c cert-reloader --tail=100 ``` Remediation: @@ -327,12 +466,12 @@ Common causes: ## Scale Down and Decommission ```bash -kubectl -n exec -c cockroachdb -- \ +kubectl -n "$CRDB_NAMESPACE" exec -c cockroachdb -- \ /cockroach/cockroach node status --decommission -kubectl -n get crdbnodes -o json | jq '[.items[] | select(.status.phase=="Decommissioning")] | {count: length, nodes: [.[].metadata.name]}' +kubectl -n "$CRDB_NAMESPACE" get crdbnodes -o json | jq '[.items[] | select(.status.decommission != null and .status.decommission != "")] | {count: length, nodes: [.[] | {name: .metadata.name, decommission: .status.decommission}]}' -kubectl -n logs -l app=cockroach-operator --tail=300 | grep -Ei 'decommission|drain|scale|blocking_ranges' || true +kubectl -n "$OPERATOR_NAMESPACE" logs -l app=cockroach-operator --tail=300 | grep -Ei 'decommission|drain|scale|blocking_ranges' || true ``` Questions to answer: @@ -349,22 +488,22 @@ Only use these after collecting evidence and confirming the risk with the custom Disable reconciliation for one cluster: ```bash -kubectl -n patch crdbcluster --type=merge -p '{"spec":{"mode":"Disabled"}}' +kubectl -n "$CRDB_NAMESPACE" patch crdbcluster "$CRDBCLUSTER" --type=merge -p '{"spec":{"mode":"Disabled"}}' # Resume reconciliation: -kubectl -n patch crdbcluster --type=merge -p '{"spec":{"mode":"MutableOnly"}}' +kubectl -n "$CRDB_NAMESPACE" patch crdbcluster "$CRDBCLUSTER" --type=merge -p '{"spec":{"mode":"MutableOnly"}}' ``` Restart the operator after evidence is collected: ```bash -kubectl -n rollout restart deploy/cockroach-operator +kubectl -n "$OPERATOR_NAMESPACE" rollout restart deploy/cockroach-operator ``` User-approved timestamp rolling restart: ```bash -helm -n upgrade \ +helm -n "$CRDB_NAMESPACE" upgrade "$CRDB_HELM_RELEASE" \ --reuse-values \ --set-string cockroachdb.crdbCluster.timestamp="$(date -u +%Y-%m-%dT%H:%M:%SZ)" ``` diff --git a/skills/cockroachdb-onboarding-and-migrations/debugging-cockroachdb-operator-migrations/SKILL.md b/skills/cockroachdb-onboarding-and-migrations/debugging-cockroachdb-operator-migrations/SKILL.md index d23802e9..bd199f17 100644 --- a/skills/cockroachdb-onboarding-and-migrations/debugging-cockroachdb-operator-migrations/SKILL.md +++ b/skills/cockroachdb-onboarding-and-migrations/debugging-cockroachdb-operator-migrations/SKILL.md @@ -16,7 +16,7 @@ Debugs migration from Helm StatefulSet or public operator v1alpha1 workloads to - A Helm StatefulSet to operator migration is stuck or unclear - A public operator v1alpha1 to v1beta1 migration has conversion or webhook issues - Migration labels remain `start` or `finalized` -- `status.migration.phase` or `status.migration.message` reports an error +- The schema-grounded migration status field or migration labels report an error - Source StatefulSet ownership, pod ownerReferences, or PVC ownerReferences are unclear - The operator stops reconciling after migration @@ -32,6 +32,9 @@ Debugs migration from Helm StatefulSet or public operator v1alpha1 workloads to ## Execution Discipline - Execute one step at a time and inspect the output before moving on. Migration phase, source workload state, and ownership determine which later checks are safe. +- Run every step in the same shell session. The commands rely on `OPERATOR_NAMESPACE`, `CRDB_NAMESPACE`, `CRDB_MIGRATION_DIR`, and the `CRDB_*_PATH` exports set in Step 1; a fresh shell loses those variables and later `jq`/`kubectl` calls will silently target the wrong object. +- Never infer a target `CrdbCluster` object name from a Helm release, service name, or CockroachDB image/version string. List `CrdbCluster` objects and use the exact `metadata.name`. +- Before reading migration, condition, or status fields from a `CrdbCluster`, save the live CRD YAML and derive field paths from the served CRD schema for the object's `apiVersion`. - Do not delete the source StatefulSet, patch labels, patch mode, restart the operator, run Helm upgrades, or change ownerReferences unless the user explicitly approves the action for the target cluster. - Do not run interactive `kubectl exec` shells or debug containers unless the user approves them and the customer policy allows the image/source. - In production or when the migration phase is ambiguous, involve TSE or the operator team before changing source or target resources. @@ -40,7 +43,7 @@ Debugs migration from Helm StatefulSet or public operator v1alpha1 workloads to - Migration type: Helm StatefulSet to operator, or public operator v1alpha1 to v1beta1 - Source resource name and namespace -- Target `CrdbCluster` name and namespace +- Target namespace and discovered `CrdbCluster.metadata.name` - Operator namespace and version - Values file or migration command used - Current migration label and phase @@ -49,9 +52,74 @@ Debugs migration from Helm StatefulSet or public operator v1alpha1 workloads to ## Step 1: Confirm Migration Documentation and Versions ```bash -kubectl -n get deploy cockroach-operator -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' +export OPERATOR_NAMESPACE="" +export CRDB_NAMESPACE="" +export CRDB_HELM_RELEASE="" +export CRDB_MIGRATION_DIR="${CRDB_MIGRATION_DIR:-$(mktemp -d)}" + +kubectl -n "$OPERATOR_NAMESPACE" get deploy cockroach-operator -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' kubectl get crd crdbclusters.crdb.cockroachlabs.com -o jsonpath='{.spec.versions[*].name}{"\n"}' -helm -n history || true +kubectl get crd crdbclusters.crdb.cockroachlabs.com -o yaml > "$CRDB_MIGRATION_DIR/crdbclusters-crd.yaml" +kubectl get crd crdbclusters.crdb.cockroachlabs.com -o json > "$CRDB_MIGRATION_DIR/crdbclusters-crd.json" +helm -n "$CRDB_NAMESPACE" history "$CRDB_HELM_RELEASE" || true + +kubectl -n "$CRDB_NAMESPACE" get crdbcluster -o json | jq -r ' + .items[] + | [.metadata.name, .apiVersion, (.metadata.labels["app.kubernetes.io/instance"] // ""), (.metadata.labels["crdb.io/migrate"] // ""), (.metadata.labels["crdb.cockroachlabs.com/migration"] // "")] + | @tsv +' +``` + +If no `CrdbCluster` rows are returned, stop the object-specific migration debugging and report that no live `CrdbCluster` exists in the namespace. You may collect `CrdbNode` owner references and labels as teardown evidence, but do not treat those values as a replacement for a discovered `CrdbCluster`. + +If multiple rows are returned, choose the target by `metadata.name`; do not use a CockroachDB version or image tag as the object name. + +```bash +export CRDBCLUSTER="" +test -n "$CRDBCLUSTER" + +kubectl -n "$CRDB_NAMESPACE" get crdbcluster "$CRDBCLUSTER" -o yaml > "$CRDB_MIGRATION_DIR/crdbcluster.yaml" +kubectl -n "$CRDB_NAMESPACE" get crdbcluster "$CRDBCLUSTER" -o json > "$CRDB_MIGRATION_DIR/crdbcluster.json" + +export CRDBCLUSTER_API_VERSION="$(jq -r '.apiVersion | split("/")[-1]' "$CRDB_MIGRATION_DIR/crdbcluster.json")" +export CRDBCLUSTER_SCHEMA_JSON="$CRDB_MIGRATION_DIR/crdbcluster-schema.json" +jq -e --arg version "$CRDBCLUSTER_API_VERSION" ' + .spec.versions[] | select(.name == $version) | .schema.openAPIV3Schema +' "$CRDB_MIGRATION_DIR/crdbclusters-crd.json" > "$CRDBCLUSTER_SCHEMA_JSON" + +crdb_schema_has() { + jq -e --arg path "$1" ' + def has_schema_path($schema; $parts): + if ($parts | length) == 0 then true + elif (($schema.properties? // {}) | has($parts[0])) then + has_schema_path($schema.properties[$parts[0]]; $parts[1:]) + else false + end; + has_schema_path(.; $path | split(".")) + ' "$CRDBCLUSTER_SCHEMA_JSON" >/dev/null +} + +crdb_first_schema_path() { + for schema_path in "$@"; do + if crdb_schema_has "$schema_path"; then + printf '%s\n' "$schema_path" + return 0 + fi + done + printf '\n' +} + +export CRDB_MODE_PATH="$(crdb_first_schema_path spec.mode)" +export CRDB_MIGRATION_PATH="$(crdb_first_schema_path status.migration)" +export CRDB_CONDITIONS_PATH="$(crdb_first_schema_path status.conditions)" +export CRDB_OBSERVED_GENERATION_PATH="$(crdb_first_schema_path status.observedGeneration)" + +printf '%s\n' \ + "apiVersion=$CRDBCLUSTER_API_VERSION" \ + "mode=$CRDB_MODE_PATH" \ + "migration=$CRDB_MIGRATION_PATH" \ + "conditions=$CRDB_CONDITIONS_PATH" \ + "observedGeneration=$CRDB_OBSERVED_GENERATION_PATH" ``` Check the local migration guide and chart changelog for version-specific migration fixes: @@ -62,20 +130,34 @@ Check the local migration guide and chart changelog for version-specific migrati ## Step 2: Inspect Migration State ```bash -kubectl -n get crdbcluster -o json | jq '{ - mode: .spec.mode, +jq \ + --arg modePath "$CRDB_MODE_PATH" \ + --arg migrationPath "$CRDB_MIGRATION_PATH" \ + --arg conditionsPath "$CRDB_CONDITIONS_PATH" \ + --arg observedGenerationPath "$CRDB_OBSERVED_GENERATION_PATH" \ + ' + def value($path): if $path == "" then null else getpath($path | split(".")) end; + { + apiVersion, + name: .metadata.name, + schemaPaths: { + mode: $modePath, + migration: $migrationPath, + conditions: $conditionsPath, + observedGeneration: $observedGenerationPath + }, + mode: value($modePath), migrateLabel: .metadata.labels["crdb.io/migrate"], migrationStatus: .metadata.labels["crdb.cockroachlabs.com/migration"], - migrationPhase: .status.migration.phase, - migrationMessage: .status.migration.message, - initialized: [.status.conditions[]? | select(.type=="Initialized" or .type=="ClusterInitialized")], + migration: value($migrationPath), + conditions: value($conditionsPath), generation: .metadata.generation, - observedGeneration: .status.observedGeneration -}' + observedGeneration: value($observedGenerationPath) +}' "$CRDB_MIGRATION_DIR/crdbcluster.json" -kubectl -n logs -l app=cockroach-operator --tail=300 | grep -Ei 'migrationctrl|migration|phase|cert' || true -kubectl -n get crdbnodes -o wide -kubectl -n get crdbnodes -o yaml +kubectl -n "$OPERATOR_NAMESPACE" logs -l app=cockroach-operator --tail=300 | grep -Ei 'migrationctrl|migration|phase|cert' || true +kubectl -n "$CRDB_NAMESPACE" get crdbnodes -o wide +kubectl -n "$CRDB_NAMESPACE" get crdbnodes -o yaml ``` ## Step 3: Inspect Source Workload @@ -83,25 +165,25 @@ kubectl -n get crdbnodes -o yaml For Helm StatefulSet migration: ```bash -kubectl -n get sts -o json | jq '{ +kubectl -n "$CRDB_NAMESPACE" get sts -o json | jq '{ replicas: .spec.replicas, readyReplicas: .status.readyReplicas, migrateLabel: .metadata.labels["crdb.io/migrate"], ownerReferences: .metadata.ownerReferences }' -kubectl -n get sts -o yaml -kubectl -n get pods -l app.kubernetes.io/name=cockroachdb -o yaml | grep -E 'name:|ownerReferences:|kind:|uid:|controller:' -A8 -kubectl -n get pvc -o yaml | grep -E 'name:|ownerReferences:|kind:|uid:|controller:' -A8 +kubectl -n "$CRDB_NAMESPACE" get sts -o yaml +kubectl -n "$CRDB_NAMESPACE" get pods -l app.kubernetes.io/name=cockroachdb -o yaml | grep -E 'name:|ownerReferences:|kind:|uid:|controller:' -A8 +kubectl -n "$CRDB_NAMESPACE" get pvc -o yaml | grep -E 'name:|ownerReferences:|kind:|uid:|controller:' -A8 ``` For public operator v1alpha1 migration: ```bash -kubectl -n get crdbcluster.crdb.cockroachlabs.com -o yaml -kubectl -n get crdbcluster.v1beta1.crdb.cockroachlabs.com -o yaml 2>&1 || true -kubectl -n get svc cockroach-webhook-service -kubectl -n get endpoints cockroach-webhook-service +kubectl -n "$CRDB_NAMESPACE" get crdbcluster.crdb.cockroachlabs.com "$CRDBCLUSTER" -o yaml +kubectl -n "$CRDB_NAMESPACE" get crdbcluster.v1beta1.crdb.cockroachlabs.com "$CRDBCLUSTER" -o yaml 2>&1 || true +kubectl -n "$OPERATOR_NAMESPACE" get svc cockroach-webhook-service +kubectl -n "$OPERATOR_NAMESPACE" get endpoints cockroach-webhook-service kubectl get validatingwebhookconfigurations | grep cockroach ``` @@ -122,9 +204,12 @@ The operator supports both v1alpha1 and v1beta1 through conversion webhooks. If ### Migration Not Progressing ```bash -kubectl -n get crdbcluster -o jsonpath='{.status.migration}{"\n"}' -kubectl -n logs -l app=cockroach-operator --tail=300 | grep -Ei 'migration|phase|cert|error' || true -kubectl -n get deploy cockroach-operator -o jsonpath='{.spec.template.spec.containers[0].args}{"\n"}' +jq --arg migrationPath "$CRDB_MIGRATION_PATH" ' + def value($path): if $path == "" then null else getpath($path | split(".")) end; + {migrationPath: $migrationPath, migration: value($migrationPath)} +' "$CRDB_MIGRATION_DIR/crdbcluster.json" +kubectl -n "$OPERATOR_NAMESPACE" logs -l app=cockroach-operator --tail=300 | grep -Ei 'migration|phase|cert|error' || true +kubectl -n "$OPERATOR_NAMESPACE" get deploy cockroach-operator -o jsonpath='{.spec.template.spec.containers[0].args}{"\n"}' ``` Check: @@ -148,10 +233,10 @@ Use [configuring-cockroachdb-helm-tls](../../cockroachdb-operations-and-lifecycl ### PodMigration Issues ```bash -kubectl -n get pods -l app.kubernetes.io/name=cockroachdb -o wide -kubectl -n describe pods -l app.kubernetes.io/name=cockroachdb -kubectl -n get crdbnodes -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,NODE_ID:.status.nodeID,OBSERVED:.status.observedGeneration -kubectl -n get pvc -o json | jq '.items[] | {name: .metadata.name, ownerReferences: .metadata.ownerReferences, storageClass: .spec.storageClassName, capacity: .status.capacity}' +kubectl -n "$CRDB_NAMESPACE" get pods -l app.kubernetes.io/name=cockroachdb -o wide +kubectl -n "$CRDB_NAMESPACE" describe pods -l app.kubernetes.io/name=cockroachdb +kubectl -n "$CRDB_NAMESPACE" get crdbnodes -o 'custom-columns=NAME:.metadata.name,DECOMMISSION:.status.decommission,NODE_ID:.status.nodeID,OBSERVED:.status.observedGeneration' +kubectl -n "$CRDB_NAMESPACE" get pvc -o json | jq '.items[] | {name: .metadata.name, ownerReferences: .metadata.ownerReferences, storageClass: .spec.storageClassName, capacity: .status.capacity}' ``` Check: @@ -185,7 +270,7 @@ Return findings in this order: 1. Migration type and current phase 2. Source workload state -3. Target `CrdbCluster` mode, migration labels, and migration message +3. Target `CrdbCluster` mode, migration labels, and schema-grounded migration status 4. `CrdbNode`, pod, and PVC ownership state 5. Current blocker and likely cause 6. Safe next action diff --git a/skills/cockroachdb-onboarding-and-migrations/installing-cockroachdb-with-helm/SKILL.md b/skills/cockroachdb-onboarding-and-migrations/installing-cockroachdb-with-helm/SKILL.md index 3d7d63c1..8db10d7a 100644 --- a/skills/cockroachdb-onboarding-and-migrations/installing-cockroachdb-with-helm/SKILL.md +++ b/skills/cockroachdb-onboarding-and-migrations/installing-cockroachdb-with-helm/SKILL.md @@ -44,6 +44,9 @@ Collect these before changing the cluster: ## Execution Discipline - Execute one step at a time and inspect the output before moving on. Preflight output determines whether installation can continue. +- Run every step in the same shell session. Verification depends on `CRDB_NAMESPACE`, `CRDB_VERIFY_DIR`, and the `CRDB_*_PATH` exports set alongside the CRD schema helpers; a new shell drops them and downstream `jq` calls will emit empty or wrong values. +- During verification, never infer a `CrdbCluster` object name from the Helm release or CockroachDB image/version. List `CrdbCluster` objects and use the exact `metadata.name`. +- Before reading `CrdbCluster` status fields, save the live CRD YAML and derive field paths from the served CRD schema for the object's `apiVersion`. - Do not run `helm install`, `helm upgrade`, `kubectl apply`, or any mutating command unless the user explicitly approves it for the target Kubernetes context and namespace. - Stop before installing if cluster-scoped RBAC, node labels, storage class, TLS mode, registry access, or multi-region prerequisites are unclear. - In production or restricted environments, involve TSE, the platform team, or the operator team before changing RBAC, webhook, operator, certificate, storage, or network configuration. @@ -162,18 +165,100 @@ Adjust `code`, `cloudProvider`, `namespace`, storage size, and storage class to Check Kubernetes resources: ```bash -kubectl -n cockroach-operator-system get deploy cockroach-operator -kubectl -n cockroach-operator-system logs deploy/cockroach-operator --tail=100 -kubectl -n cockroachdb get crdbcluster,crdbnode,pods,svc -kubectl -n cockroachdb get crdbcluster crdb -o jsonpath='{.status.readyNodes}{" ready, reconciled="}{.status.reconciled}{" version="}{.status.version}{"\n"}' -kubectl -n cockroachdb get crdbnodes -o custom-columns=NAME:.metadata.name,NODE_ID:.status.nodeID,CONDITIONS:.status.conditions[*].type,TOPOLOGY:.status.topologyValues +export OPERATOR_NAMESPACE=cockroach-operator-system +export CRDB_NAMESPACE=cockroachdb +export CRDB_VERIFY_DIR="${CRDB_VERIFY_DIR:-$(mktemp -d)}" + +kubectl -n "$OPERATOR_NAMESPACE" get deploy cockroach-operator +kubectl -n "$OPERATOR_NAMESPACE" logs deploy/cockroach-operator --tail=100 +kubectl -n "$CRDB_NAMESPACE" get crdbcluster,crdbnode,pods,svc + +kubectl get crd crdbclusters.crdb.cockroachlabs.com -o yaml > "$CRDB_VERIFY_DIR/crdbclusters-crd.yaml" +kubectl get crd crdbclusters.crdb.cockroachlabs.com -o json > "$CRDB_VERIFY_DIR/crdbclusters-crd.json" +kubectl -n "$CRDB_NAMESPACE" get crdbcluster -o json | jq -r ' + .items[] + | [.metadata.name, .apiVersion, (.metadata.labels["app.kubernetes.io/instance"] // ""), (.metadata.generation | tostring)] + | @tsv +' +``` + +If no `CrdbCluster` rows are returned, stop the object-specific verification and report that no live `CrdbCluster` exists in the namespace. You may collect `CrdbNode` owner references and labels as teardown evidence, but do not treat those values as a replacement for a discovered `CrdbCluster`. + +If multiple `CrdbCluster` rows are returned, choose the target by `metadata.name`; do not use the Helm release or CockroachDB version as a substitute. + +```bash +export CRDBCLUSTER="" +test -n "$CRDBCLUSTER" + +kubectl -n "$CRDB_NAMESPACE" get crdbcluster "$CRDBCLUSTER" -o yaml > "$CRDB_VERIFY_DIR/crdbcluster.yaml" +kubectl -n "$CRDB_NAMESPACE" get crdbcluster "$CRDBCLUSTER" -o json > "$CRDB_VERIFY_DIR/crdbcluster.json" + +export CRDBCLUSTER_API_VERSION="$(jq -r '.apiVersion | split("/")[-1]' "$CRDB_VERIFY_DIR/crdbcluster.json")" +export CRDBCLUSTER_SCHEMA_JSON="$CRDB_VERIFY_DIR/crdbcluster-schema.json" +jq -e --arg version "$CRDBCLUSTER_API_VERSION" ' + .spec.versions[] | select(.name == $version) | .schema.openAPIV3Schema +' "$CRDB_VERIFY_DIR/crdbclusters-crd.json" > "$CRDBCLUSTER_SCHEMA_JSON" + +crdb_schema_has() { + jq -e --arg path "$1" ' + def has_schema_path($schema; $parts): + if ($parts | length) == 0 then true + elif (($schema.properties? // {}) | has($parts[0])) then + has_schema_path($schema.properties[$parts[0]]; $parts[1:]) + else false + end; + has_schema_path(.; $path | split(".")) + ' "$CRDBCLUSTER_SCHEMA_JSON" >/dev/null +} + +crdb_first_schema_path() { + for schema_path in "$@"; do + if crdb_schema_has "$schema_path"; then + printf '%s\n' "$schema_path" + return 0 + fi + done + printf '\n' +} + +export CRDB_READY_NODES_PATH="$(crdb_first_schema_path status.readyNodes)" +export CRDB_RECONCILED_PATH="$(crdb_first_schema_path status.reconciled)" +export CRDB_STATUS_VERSION_PATH="$(crdb_first_schema_path status.version)" + +jq \ + --arg readyNodesPath "$CRDB_READY_NODES_PATH" \ + --arg reconciledPath "$CRDB_RECONCILED_PATH" \ + --arg statusVersionPath "$CRDB_STATUS_VERSION_PATH" \ + ' + def value($path): if $path == "" then null else getpath($path | split(".")) end; + { + apiVersion, + name: .metadata.name, + schemaPaths: { + readyNodes: $readyNodesPath, + reconciled: $reconciledPath, + statusVersion: $statusVersionPath + }, + readyNodes: value($readyNodesPath), + reconciled: value($reconciledPath), + statusVersion: value($statusVersionPath) + } +' "$CRDB_VERIFY_DIR/crdbcluster.json" + +kubectl -n "$CRDB_NAMESPACE" get crdbnodes -o json | jq '.items[] | { + name: .metadata.name, + nodeID: .status.nodeID, + decommission: .status.decommission, + conditions: .status.conditions, + topology: .status.topologyValues +}' ``` Expected state: - `cockroach-operator` Deployment is available. - CRDs `crdbclusters.crdb.cockroachlabs.com` and `crdbnodes.crdb.cockroachlabs.com` exist. -- `CrdbCluster.status.reconciled` is `true` and `readyNodes` equals the regional node count. +- Schema-grounded `CrdbCluster` status shows `reconciled=true` and `readyNodes` equals the regional node count when those fields exist in the served CRD version. - Each `CrdbNode` has a `nodeID` and Ready/Running conditions. - CockroachDB pods are Running and Ready. @@ -193,7 +278,7 @@ Return a concise install report: - Operator release, CockroachDB release, and chart versions - Values file path or inline values that were applied - CRD registration status -- `CrdbCluster` ready/reconciled status and node count +- Schema-grounded `CrdbCluster` ready/reconciled status and node count - SQL verification result or the exact blocker if SQL could not be verified ## Troubleshooting Handoff diff --git a/skills/cockroachdb-onboarding-and-migrations/validating-cockroachdb-helm-multiregion/SKILL.md b/skills/cockroachdb-onboarding-and-migrations/validating-cockroachdb-helm-multiregion/SKILL.md index a0410b80..0f7f795c 100644 --- a/skills/cockroachdb-onboarding-and-migrations/validating-cockroachdb-helm-multiregion/SKILL.md +++ b/skills/cockroachdb-onboarding-and-migrations/validating-cockroachdb-helm-multiregion/SKILL.md @@ -39,6 +39,7 @@ Validates the high-risk prerequisites for multi-region CockroachDB deployments m ## Execution Discipline - Execute one step at a time and inspect the output before moving on. Region inventory, DNS results, network results, and certificate state determine which later checks are relevant. +- Do not infer cross-region service names from the Helm release alone. List Services in the peer namespace and use the actual DNS service name that CockroachDB pods are expected to join. - Do not create debug pods, run interactive shells, use external diagnostic images, or perform Helm upgrades unless the user explicitly approves them for each participating cluster. - In air-gapped or private-registry environments, use customer-approved diagnostic images mirrored into the customer's registry. - In production or when cross-region networking, certificate trust, or locality is unclear, involve TSE or the operator team before adding regions or changing chart values. @@ -91,7 +92,13 @@ Checklist: ## Step 3: Validate Cross-Region DNS and Network Paths -From a temporary debugging pod in each region, verify that peer region service names resolve and connect. Replace names with the actual release and namespace. +From a temporary debugging pod in each region, verify that peer region service names resolve and connect. First discover the peer service name; do not assume it is identical to the Helm release. + +```bash +kubectl --context -n get service,endpoints -o wide +export PEER_SERVICE="" +test -n "$PEER_SERVICE" +``` ```bash kubectl --context -n run crdb-dns-check \ @@ -101,8 +108,8 @@ kubectl --context -n run crdb-dns-check \ Inside the pod: ```sh -nslookup ..svc. -nc -vz ..svc. 26258 +nslookup ..svc. +nc -vz ..svc. 26258 ``` If `nc` is unavailable, use an approved network diagnostic image or cloud-native connectivity test. Do not proceed until DNS and TCP connectivity are proven in both directions. diff --git a/skills/cockroachdb-operations-and-lifecycle/configuring-cockroachdb-helm-tls/SKILL.md b/skills/cockroachdb-operations-and-lifecycle/configuring-cockroachdb-helm-tls/SKILL.md index 749115b3..34674df3 100644 --- a/skills/cockroachdb-operations-and-lifecycle/configuring-cockroachdb-helm-tls/SKILL.md +++ b/skills/cockroachdb-operations-and-lifecycle/configuring-cockroachdb-helm-tls/SKILL.md @@ -31,6 +31,9 @@ Guides TLS configuration for operator-managed CockroachDB clusters installed wit ## Execution Discipline - Execute one step at a time and inspect the output before moving on. Certificate mode, Secret names, and issuer state determine which later checks are relevant. +- Run every step in the same shell session. Certificate discovery relies on `CRDB_NAMESPACE`, `CRDB_TLS_DIR`, and the `CRDB_CERTIFICATES_PATH` export set alongside the CRD schema helper; a fresh shell drops them and later `jq`/`kubectl` calls will produce empty output. +- For an installed cluster, never infer a `CrdbCluster` object name from the Helm release or CockroachDB image/version. List `CrdbCluster` objects and use the exact `metadata.name`. +- Before reading certificate references from a `CrdbCluster`, save the live CRD YAML and derive the certificate field path from the served CRD schema for the object's `apiVersion`. - Do not change TLS mode, replace Secrets, patch cert-manager resources, run debug containers, or perform Helm upgrades unless the user explicitly approves the action for the target cluster. - Never print private key data. Use metadata checks for expiry, issuer, subject, SANs, and required key presence. - In production or when certificate ownership is unclear, involve TSE or the operator team before rotation, regeneration, debug containers, or restart actions. @@ -184,9 +187,66 @@ State clearly that insecure mode has no TLS or authentication protections and is ## Post-Install Verification ```bash -kubectl -n get crdbcluster -o yaml | grep -A12 certificates -kubectl -n get secret,configmap | grep -E 'cockroach|crdb' -kubectl -n get pods +export CRDB_NAMESPACE="" +export CRDB_TLS_DIR="${CRDB_TLS_DIR:-$(mktemp -d)}" + +kubectl get crd crdbclusters.crdb.cockroachlabs.com -o yaml > "$CRDB_TLS_DIR/crdbclusters-crd.yaml" +kubectl get crd crdbclusters.crdb.cockroachlabs.com -o json > "$CRDB_TLS_DIR/crdbclusters-crd.json" +kubectl -n "$CRDB_NAMESPACE" get crdbcluster -o json | jq -r ' + .items[] + | [.metadata.name, .apiVersion, (.metadata.labels["app.kubernetes.io/instance"] // ""), (.metadata.generation | tostring)] + | @tsv +' +``` + +If no `CrdbCluster` rows are returned, stop the object-specific TLS validation and report that no live `CrdbCluster` exists in the namespace. You may collect `CrdbNode` owner references and labels as teardown evidence, but do not treat those values as a replacement for a discovered `CrdbCluster`. + +If multiple `CrdbCluster` rows are returned, choose the target by `metadata.name`; do not use the Helm release or CockroachDB version as a substitute. + +```bash +export CRDBCLUSTER="" +test -n "$CRDBCLUSTER" + +kubectl -n "$CRDB_NAMESPACE" get crdbcluster "$CRDBCLUSTER" -o yaml > "$CRDB_TLS_DIR/crdbcluster.yaml" +kubectl -n "$CRDB_NAMESPACE" get crdbcluster "$CRDBCLUSTER" -o json > "$CRDB_TLS_DIR/crdbcluster.json" + +export CRDBCLUSTER_API_VERSION="$(jq -r '.apiVersion | split("/")[-1]' "$CRDB_TLS_DIR/crdbcluster.json")" +export CRDBCLUSTER_SCHEMA_JSON="$CRDB_TLS_DIR/crdbcluster-schema.json" +jq -e --arg version "$CRDBCLUSTER_API_VERSION" ' + .spec.versions[] | select(.name == $version) | .schema.openAPIV3Schema +' "$CRDB_TLS_DIR/crdbclusters-crd.json" > "$CRDBCLUSTER_SCHEMA_JSON" + +crdb_schema_has() { + jq -e --arg path "$1" ' + def has_schema_path($schema; $parts): + if ($parts | length) == 0 then true + elif (($schema.properties? // {}) | has($parts[0])) then + has_schema_path($schema.properties[$parts[0]]; $parts[1:]) + else false + end; + has_schema_path(.; $path | split(".")) + ' "$CRDBCLUSTER_SCHEMA_JSON" >/dev/null +} + +crdb_first_schema_path() { + for schema_path in "$@"; do + if crdb_schema_has "$schema_path"; then + printf '%s\n' "$schema_path" + return 0 + fi + done + printf '\n' +} + +export CRDB_CERTIFICATES_PATH="$(crdb_first_schema_path spec.template.spec.certificates spec.certificates)" + +jq --arg certificatesPath "$CRDB_CERTIFICATES_PATH" ' + def value($path): if $path == "" then null else getpath($path | split(".")) end; + {apiVersion, name: .metadata.name, certificatesPath: $certificatesPath, certificates: value($certificatesPath)} +' "$CRDB_TLS_DIR/crdbcluster.json" + +kubectl -n "$CRDB_NAMESPACE" get secret,configmap | grep -E 'cockroach|crdb' +kubectl -n "$CRDB_NAMESPACE" get pods ``` For self-signer, confirm the self-signer job ran and the generated CA, node, and client resources exist. For cert-manager, confirm `Certificate` resources are Ready. For external certificates, confirm the `CrdbCluster` references the expected names. @@ -196,10 +256,13 @@ For self-signer, confirm the self-signer job ran and the generated CA, node, and Use this section when pods report `x509` errors, certificate rotation is not reflected in pods, the `cert-reloader` sidecar fails, or TSC asks for certificate evidence. Collect metadata only; do not print private keys. ```bash -kubectl -n get crdbcluster -o yaml | grep -A20 certificates -kubectl -n get secret,configmap | grep -E 'ca|node|client|tls|cert|cockroach|crdb' -kubectl -n get pod -o jsonpath='{.spec.containers[*].name}{"\n"}' -kubectl -n logs -c cert-reloader --tail=100 +jq --arg certificatesPath "$CRDB_CERTIFICATES_PATH" ' + def value($path): if $path == "" then null else getpath($path | split(".")) end; + {apiVersion, name: .metadata.name, certificatesPath: $certificatesPath, certificates: value($certificatesPath)} +' "$CRDB_TLS_DIR/crdbcluster.json" +kubectl -n "$CRDB_NAMESPACE" get secret,configmap | grep -E 'ca|node|client|tls|cert|cockroach|crdb' +kubectl -n "$CRDB_NAMESPACE" get pod -o jsonpath='{.spec.containers[*].name}{"\n"}' +kubectl -n "$CRDB_NAMESPACE" logs -c cert-reloader --tail=100 ``` Inspect the node certificate: