Cilium full steam ahead - Gateway, ClusterMesh, SPIRE and all the good things

[olga-mir]

Current status.

cluster-01 and cluster-02 are provisioned with different degree of success.

Spire fails to start due to volume oncluster-01, which otherwise looks good.

Events:
  Type    Reason         Age                  From                         Message
  ----    ------         ----                 ----                         -------
  Normal  FailedBinding  52s (x26 over 7m2s)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set
%
% k get  pvc -n cilium-spire   spire-data-spire-server-0 -o yaml | yq .                                                                                                            develop 20:25:23
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  creationTimestamp: "2024-07-09T10:17:56Z"
  finalizers:
    - kubernetes.io/pvc-protection
  labels:
    app: spire-server
  name: spire-data-spire-server-0
  namespace: cilium-spire
  resourceVersion: "421"
  uid: cf2ac766-be6b-4793-a6cf-fc5d524fcf9d
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  volumeMode: Filesystem
status:
  phase: Pending

on cluster-02 the provisioning didn't work fully, CNI didn't get up.

% k get po -A    
NAMESPACE     NAME                                                                     READY   STATUS    RESTARTS   AGE
flux-system   helm-controller-5c48c6d4c8-h86zm                                         0/1     Pending   0          11m
flux-system   kustomize-controller-5945c46b66-pl4s7                                    0/1     Pending   0          11m
flux-system   notification-controller-77f747f7c4-zsxwv                                 0/1     Pending   0          11m
flux-system   source-controller-645749f7d4-qsj5r                                       0/1     Pending   0          11m
kube-system   coredns-565d847f94-7blx4                                                 0/1     Pending   0          11m
kube-system   coredns-565d847f94-gvpmz                                                 0/1     Pending   0          11m
kube-system   etcd-ip-10-0-175-38.ap-southeast-2.compute.internal                      1/1     Running   0          11m
kube-system   kube-apiserver-ip-10-0-175-38.ap-southeast-2.compute.internal            1/1     Running   0          11m
kube-system   kube-controller-manager-ip-10-0-175-38.ap-southeast-2.compute.internal   1/1     Running   0          11m
kube-system   kube-scheduler-ip-10-0-175-38.ap-southeast-2.compute.internal            1/1     Running   0          11m

nodes:

% k get no                                                                                                                                                                         develop 20:29:23
NAME                                             STATUS     ROLES           AGE   VERSION
ip-10-0-170-89.ap-southeast-2.compute.internal   NotReady   <none>          11m   v1.25.4
ip-10-0-175-38.ap-southeast-2.compute.internal   NotReady   control-plane   12m   v1.25.4

on perm mgmt cluster:

k get clusters -A                                                                                                                                                                develop 20:31:21
NAMESPACE      NAME           CLUSTERCLASS   PHASE         AGE   VERSION
cluster-01     cluster-01                    Provisioned   18m
cluster-02     cluster-02                    Provisioned   18m
cluster-mgmt   cluster-mgmt                  Provisioned   25m

and

 % flux get all -A                                                                                                                                                                  develop 20:31:45
NAMESPACE       NAME                            REVISION                SUSPENDED       READY   MESSAGE
flux-system     gitrepository/flux-system       develop@sha1:efd7f646   False           True    stored artifact for revision 'develop@sha1:efd7f646'

NAMESPACE       NAME                            REVISION                SUSPENDED       READY   MESSAGE
cluster-01      kustomization/flux-remote       develop@sha1:efd7f646   False           True    Applied revision: develop@sha1:efd7f646
cluster-02      kustomization/flux-remote       develop@sha1:efd7f646   False           True    Applied revision: develop@sha1:efd7f646
flux-system     kustomization/caaph             develop@sha1:efd7f646   False           True    Applied revision: develop@sha1:efd7f646
flux-system     kustomization/caaph-cni-01      develop@sha1:efd7f646   False           True    Applied revision: develop@sha1:efd7f646
flux-system     kustomization/caaph-cni-02      develop@sha1:efd7f646   False           True    Applied revision: develop@sha1:efd7f646
flux-system     kustomization/flux-system       develop@sha1:efd7f646   False           True    Applied revision: develop@sha1:efd7f646

cluster-02 doesn't have CNI even though hcp looks successful

% k get helmchartproxies -A     
NAMESPACE    NAME                  READY   REASON
cluster-01   cilium-cluster-mesh   True
cluster-02   cilium-cluster-mesh   True

Another problem is wait in go code, throws error but it should be working it looks like it is pointing to the correct cluster and the cluster is in the namespace but can't still timeout.

go % task run-deploy                                                                                                                                                                                 develop 19:53:29
task: [build-app] go build ./cmd/multicluster-demo
task: [run-deploy] ./multicluster-demo deploy --config .
2024-07-09T19:53:45+10:00       INFO    Create `kind` cluster
2024-07-09T19:54:30+10:00       INFO    Kind cluster is ready
2024-07-09T19:54:30+10:00       INFO    Installing Cluster API on `kind` cluster
2024-07-09T19:54:30+10:00       INFO    Creating Cluster API clients for kube context   {"name": "kind-tmp-mgmt"}
2024-07-09T19:54:30+10:00       INFO    Initializing Cluster API        {"cluster": "tmp-mgmt", "initoptions": {"Kubeconfig":{"Path":"/Users/olga/.kube/config","Context":"kind-tmp-mgmt"},"CoreProvider":"","Bootstra
pProviders":null,"InfrastructureProviders":["aws:v2.3.1"],"ControlPlaneProviders":null,"IPAMProviders":null,"RuntimeExtensionProviders":null,"AddonProviders":null,"TargetNamespace":"","LogUsageInstructions":false,"
WaitProviders":false,"WaitProviderTimeout":0,"IgnoreValidationErrors":false}}
2024-07-09T19:55:01+10:00       INFO    Installing FluxCD on `kind` cluster
2024-07-09T19:55:01+10:00       INFO    Applying gotk-components
2024-07-09T19:55:05+10:00       INFO    Waiting for Flux CRDs to become established
2024-07-09T19:55:25+10:00       INFO    Creating secret for Flux
2024-07-09T19:55:25+10:00       INFO    KubeAPIWarningLogger    v1beta1 GitRepository is deprecated, upgrade to v1
2024-07-09T19:55:25+10:00       INFO    Sleeping for 3 min to allow Flux to apply resources from the repository
2024-07-09T19:58:25+10:00       INFO    Waiting for all Flux resources to become Ready
2024-07-09T20:04:05+10:00       INFO    Wating for CAPI cluster to be provisioned and all system components healthy     {"cluster": "cluster-mgmt"}
2024-07-09T20:04:05+10:00       INFO    Wait for CAAPH resources to be Ready
2024-07-09T20:04:15+10:00       INFO    GetClusterAuthInfo for workload cluster {"name": "cluster-mgmt", "options": {"Kubeconfig":{"Path":"/Users/olga/.kube/config","Context":"kind-tmp-mgmt"},"Namespace":"cluster-m
gmt","WorkloadClusterName":"cluster-mgmt"}}
2024-07-09T20:04:15+10:00       INFO    Suspended kustomization {"name": "flux-system", "namespace": "flux-system"}
2024-07-09T20:04:15+10:00       INFO    Creating Cluster API clients for kube context   {"name": "cluster-mgmt-admin@cluster-mgmt"}
2024-07-09T20:04:15+10:00       INFO    Installing Cluster API on the permanent management cluster
2024-07-09T20:04:15+10:00       INFO    Initializing Cluster API        {"cluster": "cluster-mgmt", "initoptions": {"Kubeconfig":{"Path":"/Users/olga/.kube/config","Context":"cluster-mgmt-admin@cluster-mgmt"},"Core
Provider":"","BootstrapProviders":null,"InfrastructureProviders":["aws:v2.3.1"],"ControlPlaneProviders":null,"IPAMProviders":null,"RuntimeExtensionProviders":null,"AddonProviders":null,"TargetNamespace":"","LogUsag
eInstructions":false,"WaitProviders":false,"WaitProviderTimeout":0,"IgnoreValidationErrors":false}}
2024-07-09T20:05:50+10:00       INFO    Pivoting management cluster     {"fromContextName": "kind-tmp-mgmt", "toContextName": "cluster-mgmt-admin@cluster-mgmt"}
2024-07-09T20:06:47+10:00       INFO    Successfully pivoted Cluster API components and custom resource is available in the target cluster
2024-07-09T20:06:47+10:00       INFO    Creating FluxCD instance for permanent management cluster
2024-07-09T20:06:47+10:00       INFO    Creating secret for Flux
2024-07-09T20:06:47+10:00       INFO    Flux Secret provisioned, start checking the clusters
2024-07-09T20:06:47+10:00       INFO    GGG WaitForAllClustersProvisioning      {"cluster context name": "cluster-mgmt-admin@cluster-mgmt"}
2024-07-09T20:06:47+10:00       INFO    GGG WaitForAllClustersProvisioning In Go routine        {"namespace": "cluster-mgmt", "cluster context name": "cluster-mgmt-admin@cluster-mgmt"}
2024-07-09T20:06:47+10:00       INFO    GGG WaitForAllClustersProvisioning      {"err": "error in namespace cluster-mgmt: clusters.cluster.x-k8s.io \"cluster-mgmt\" not found"}
Error waiting for clusters to be provisioned: error in namespace cluster-mgmt: clusters.cluster.x-k8s.io "cluster-mgmt" not found