From 790f1aec0c6bde664f35b323c2229601f709ef32 Mon Sep 17 00:00:00 2001 From: clyi Date: Wed, 12 Aug 2026 13:45:55 +0800 Subject: [PATCH 1/4] docs(acp): add nodelocaldns S2 workarounds --- .../acp/NodeLocal_DNSCache_S2_Workarounds.md | 359 ++++++++++++++++++ .../acp/NodeLocal_DNSCache_S2_Workarounds.md | 359 ++++++++++++++++++ 2 files changed, 718 insertions(+) create mode 100644 docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md create mode 100644 docs/zh/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md diff --git a/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md b/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md new file mode 100644 index 000000000..ef116d3aa --- /dev/null +++ b/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md @@ -0,0 +1,359 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - '4.2.x,4.3.x,4.4.x' +--- + +# S2 Temporary Workarounds for NodeLocal DNSCache Port 8080 Conflicts and DNS Single-Point Risk + +## Problem + +After the NodeLocal DNSCache plugin is installed in an ACP cluster, the following operational risks may occur: + +- The `node-cache` Pod on each node runs with `hostNetwork: true` and listens on `127.0.0.1:8080` as its health check endpoint. If a business process, operations agent, or other node-level component also needs to use node-local port `8080`, a port conflict may occur. +- After NodeLocal DNSCache takes effect, newly created Pods use the node-local DNS address by default. If the `node-cache` Pod on a node is unavailable, DNS resolution for Pods on that node may fail. + +## Root Cause + +The current NodeLocal DNSCache plugin does not expose the following settings in the installation parameters: + +- Health check port. +- Multiple DNS server configuration for kubelet `cluster-dns`. + +The generated Corefile and DaemonSet probe use `8080` by default: + +```text +health 127.0.0.1:8080 +``` + +```yaml +livenessProbe: + httpGet: + host: 127.0.0.1 + path: /health + port: 8080 +``` + +The plugin installation job also configures kubelet `--cluster-dns` to the NodeLocal DNSCache IP. If CoreDNS ClusterIP must be used as an additional DNS server, S2 or implementation engineers need to temporarily modify the kubelet configuration on the target cluster. + +## Temporary Workarounds + +This article provides two independent S2 temporary workarounds: + +- Workaround 1: Change the NodeLocal DNSCache health check port to avoid node port `8080` conflicts. +- Workaround 2: Configure multiple DNS servers and use CoreDNS ClusterIP as an additional DNS server to reduce the impact when node-local DNS is unavailable. + +These workarounds are not persistent product capabilities. Manual changes may be overwritten after plugin upgrade, plugin reinstall, platform reconciliation, chart re-rendering, or node rebuild. For long-term use, the required configuration should be productized. + +## Workaround 1: Avoid NodeLocal DNSCache Port 8080 Conflicts + +Use this workaround when a business process, operations agent, or other node-level component must use node-local port `8080`. + +Before you start, confirm that: + +- You have an administrator kubeconfig for the target cluster. +- The business side really needs to release node-local port `8080`. +- You have selected a new port that is not used by other node components, for example `18080`. +- You have scheduled a change window, and a short rolling restart of NodeLocal DNSCache Pods is acceptable. + +### 1.1 Confirm NodeLocal DNSCache Resource Names + +Find the NodeLocal DNSCache DaemonSet and ConfigMap in the target cluster: + +```bash +kubectl get ds -A | grep -i node-local +kubectl get cm -A | grep -i node-local +``` + +Record the actual namespace, DaemonSet name, and ConfigMap name. The following steps use the default names as examples: + +```bash +NS=kube-system +DS=node-local-dns +CM=node-local-dns +NEW_PORT=18080 +``` + +If the resource names are different in the actual environment, replace these variables. + +### 1.2 Back Up Current Resources + +Back up the ConfigMap and DaemonSet before making changes: + +```bash +kubectl -n "$NS" get cm "$CM" -o yaml > node-local-dns-cm.backup.yaml +kubectl -n "$NS" get ds "$DS" -o yaml > node-local-dns-ds.backup.yaml +``` + +Confirm that the backup files are generated: + +```bash +ls -l node-local-dns-cm.backup.yaml node-local-dns-ds.backup.yaml +``` + +### 1.3 Change the Corefile Health Check Port + +Edit the NodeLocal DNSCache ConfigMap: + +```bash +kubectl -n "$NS" edit cm "$CM" +``` + +Change the health check port in the Corefile from `8080` to the new port. For example, change: + +```text +health 127.0.0.1:8080 +``` + +to: + +```text +health 127.0.0.1:18080 +``` + +Do not change the DNS service port `53` or the metrics port `9353`. + +### 1.4 Change the DaemonSet Probe Port + +Edit the NodeLocal DNSCache DaemonSet: + +```bash +kubectl -n "$NS" edit ds "$DS" +``` + +Change `livenessProbe.httpGet.port` of the `node-cache` container from `8080` to the same new port: + +```yaml +livenessProbe: + httpGet: + host: 127.0.0.1 + path: /health + port: 18080 +``` + +If the DaemonSet in the target environment also includes a `readinessProbe`, and that probe also accesses `/health` or port `8080`, change it to the same new port. + +### 1.5 Wait for the DaemonSet Rolling Update + +After the DaemonSet Pod template is changed, Kubernetes starts a DaemonSet rolling update. Wait for the update to complete: + +```bash +kubectl -n "$NS" rollout status ds "$DS" +``` + +Confirm that all NodeLocal DNSCache Pods are running: + +```bash +kubectl -n "$NS" get pods -l k8s-app="$DS" -o wide +``` + +If the Pod label in the environment is not `k8s-app=`, adjust the query label according to `.spec.selector.matchLabels` of the DaemonSet. + +### 1.6 Verify the Change + +Confirm that the ConfigMap uses the new port: + +```bash +kubectl -n "$NS" get cm "$CM" -o yaml | grep 'health 127.0.0.1' +``` + +The expected output should contain the new port, for example: + +```text +health 127.0.0.1:18080 +``` + +Confirm that the DaemonSet probe port uses the new port: + +```bash +kubectl -n "$NS" get ds "$DS" -o yaml | grep -A5 -E 'livenessProbe|readinessProbe' +``` + +The expected output should show the new probe port. + +If you need to check node port usage, log in to a node that has a NodeLocal DNSCache Pod and run: + +```bash +ss -ltnp | grep ':18080' +ss -ltnp | grep ':8080' +``` + +Expected results: + +- Port `18080` is listened on by the NodeLocal DNSCache process. +- Port `8080` is no longer listened on by the NodeLocal DNSCache process. + +Finally, verify DNS resolution from a business Pod or a temporary Pod: + +```bash +kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default.svc +``` + +The command should resolve `kubernetes.default.svc` successfully. + +### 1.7 Roll Back + +If the NodeLocal DNSCache Pod becomes abnormal or DNS resolution fails after the change, restore the backup files: + +```bash +kubectl apply -f node-local-dns-cm.backup.yaml +kubectl apply -f node-local-dns-ds.backup.yaml +kubectl -n "$NS" rollout status ds "$DS" +``` + +After rollback, verify Pod status and DNS resolution again: + +```bash +kubectl -n "$NS" get pods -l k8s-app="$DS" -o wide +kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default.svc +``` + +## Workaround 2: Configure CoreDNS ClusterIP as an Additional DNS Server + +Use this workaround when you want to reduce the single-point impact of NodeLocal DNSCache. After configuration, newly created Pods have both the NodeLocal DNSCache IP and CoreDNS ClusterIP in `/etc/resolv.conf`. + +This workaround does not guarantee transparent failover. DNS resolver retry behavior differs between business images. When the first DNS server is unavailable, some workloads may wait for timeout before trying the next DNS server, which can slow down DNS resolution during the failure. + +Before you start, confirm that: + +- You have an administrator kubeconfig for the target cluster. +- You have confirmed the NodeLocal DNSCache IP, for example `169.254.20.10`. +- You have confirmed the CoreDNS Service ClusterIP. +- You have scheduled a change window. Changing kubelet configuration requires restarting kubelet, and existing Pods do not automatically update `/etc/resolv.conf`; affected Pods need to be recreated. + +### 2.1 Get CoreDNS ClusterIP + +Query the DNS Service in the `kube-system` namespace: + +```bash +kubectl -n kube-system get svc kube-dns +``` + +Record the value in the `CLUSTER-IP` column. The following steps use `10.96.0.10` as an example; replace it with the actual value. + +If the DNS Service in the target cluster is not named `kube-dns`, find the actual name first: + +```bash +kubectl -n kube-system get svc | grep -E 'kube-dns|coredns' +``` + +### 2.2 Change kubelet cluster-dns on Nodes + +Log in to each node that needs the change, and back up the kubelet argument file: + +```bash +sudo cp -a /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/kubeadm-flags.env.bak.$(date +%Y%m%d%H%M%S) +``` + +Edit the kubelet argument file: + +```bash +sudo vi /var/lib/kubelet/kubeadm-flags.env +``` + +Change kubelet `--cluster-dns` from the single NodeLocal DNSCache IP to a combination of NodeLocal DNSCache IP and CoreDNS ClusterIP. For example, change: + +```text +--cluster-dns=169.254.20.10 +``` + +to: + +```text +--cluster-dns=169.254.20.10,10.96.0.10 +``` + +Where: + +- `169.254.20.10` is the NodeLocal DNSCache IP. +- `10.96.0.10` is the CoreDNS ClusterIP. + +Do not remove other kubelet arguments on the same line. + +### 2.3 Restart kubelet + +After saving the configuration, restart kubelet: + +```bash +sudo systemctl restart kubelet +``` + +If the target operating system does not use systemd, use the kubelet restart method supported by that environment. + +Confirm that kubelet is running again: + +```bash +sudo systemctl status kubelet +``` + +### 2.4 Recreate Affected Pods + +The kubelet `cluster-dns` change only affects newly created Pods. Existing Pods do not automatically update `/etc/resolv.conf`. + +During the change window, recreate the business Pods that need to use multiple DNS servers. The recreation method depends on the workload controller type. For example, for a Deployment: + +```bash +kubectl -n rollout restart deployment/ +kubectl -n rollout status deployment/ +``` + +### 2.5 Verify the Change + +Create a temporary Pod and confirm that `/etc/resolv.conf` contains both the NodeLocal DNSCache IP and CoreDNS ClusterIP: + +```bash +kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- cat /etc/resolv.conf +``` + +Expected output contains similar entries: + +```text +nameserver 169.254.20.10 +nameserver 10.96.0.10 +``` + +Verify DNS resolution: + +```bash +kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default.svc +``` + +If NetworkPolicy is enabled in the cluster, allow Pods to access both the NodeLocal DNSCache IP and CoreDNS ClusterIP on TCP/UDP port `53`. + +### 2.6 Persist the Configuration for Node Rebuild Scenarios + +If the cluster is upgraded by rebuilding nodes, directly changing `/var/lib/kubelet/kubeadm-flags.env` on nodes is lost after node rebuild. You need to synchronize the same multi-address `cluster-dns` value to every `kubeletExtraArgs` location in the cluster template: + +- `KubeadmControlPlane` → `initConfiguration` → `nodeRegistration` → `kubeletExtraArgs` +- `KubeadmControlPlane` → `joinConfiguration` → `nodeRegistration` → `kubeletExtraArgs` +- `KubeadmConfigTemplate` → `template` → `spec` → `joinConfiguration` → `nodeRegistration` → `kubeletExtraArgs` + +Example: + +```yaml +cluster-dns: "169.254.20.10,10.96.0.10" +``` + +### 2.7 Roll Back + +If problems occur after configuring multiple DNS servers, log in to the modified nodes and restore `/var/lib/kubelet/kubeadm-flags.env` from the backup file: + +```bash +sudo cp -a /var/lib/kubelet/kubeadm-flags.env.bak. /var/lib/kubelet/kubeadm-flags.env +sudo systemctl restart kubelet +``` + +Then recreate the affected Pods so their `/etc/resolv.conf` is regenerated. + +## Long-Term Recommendation + +The long-term fix should be implemented on the product side. For example: + +- Expose the health check port in the NodeLocal DNSCache plugin parameters, and render the value to both the Corefile and DaemonSet probe. +- Support multiple kubelet `cluster-dns` addresses in the NodeLocal DNSCache plugin or cluster configuration. +- Evaluate whether NodeLocal DNSCache should continue to be enabled by default in future DNS architecture evolution. + +Before the productized solution is delivered, this article should only be used as a temporary workaround guide for S2 or implementation engineers. diff --git a/docs/zh/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md b/docs/zh/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md new file mode 100644 index 000000000..b7693949c --- /dev/null +++ b/docs/zh/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md @@ -0,0 +1,359 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - '4.2.x,4.3.x,4.4.x' +--- + +# NodeLocal DNSCache 8080 端口冲突和 DNS 单点风险的 S2 临时规避方案 + +## 问题 + +在 ACP 集群中安装 NodeLocal DNSCache 插件后，可能遇到以下两个运维风险： + +- 每个节点上的 `node-cache` Pod 使用 `hostNetwork: true` 运行，并在节点本地监听 `127.0.0.1:8080` 作为健康检查端口。如果业务进程、运维代理或其他节点级组件也需要使用节点本地 `8080` 端口，可能出现端口冲突。 +- NodeLocal DNSCache 生效后，新建 Pod 默认使用节点本地 DNS 地址。当某个节点上的 `node-cache` Pod 不可用时，该节点上 Pod 的 DNS 解析可能失败。 + +## 根本原因 + +当前 NodeLocal DNSCache 插件未在安装参数中暴露以下配置： + +- 健康检查端口。 +- kubelet `cluster-dns` 的多 DNS server 配置。 + +插件生成的资源中，Corefile 和 DaemonSet 探针默认使用 `8080`： + +```text +health 127.0.0.1:8080 +``` + +```yaml +livenessProbe: + httpGet: + host: 127.0.0.1 + path: /health + port: 8080 +``` + +同时，插件安装任务会把 kubelet `--cluster-dns` 配置为 NodeLocal DNSCache IP。若需要把 CoreDNS ClusterIP 作为辅助 DNS server，需要由 S2 或实施人员在目标集群中临时修改 kubelet 配置。 + +## 临时方案 + +本文包含两个可独立执行的 S2 临时方案： + +- 方案一：修改 NodeLocal DNSCache 健康检查端口，规避节点 `8080` 端口冲突。 +- 方案二：配置多个 DNS server，将 CoreDNS ClusterIP 作为辅助 DNS，降低节点本地 DNS 不可用时的影响。 + +这些方案不是持久化产品能力。插件升级、重装、平台调谐、重新渲染 chart 或节点重建后，手工修改可能被覆盖。如需长期使用，应推动产品化支持。 + +## 方案一：规避 NodeLocal DNSCache 占用节点 8080 端口 + +该方案适用于业务进程、运维代理或其他节点级组件必须使用节点本地 `8080` 端口的场景。 + +操作前请确认： + +- 已获得目标集群的管理员 kubeconfig。 +- 已确认业务侧确实需要释放节点本地 `8080` 端口。 +- 已选择一个未被节点上其他组件占用的新端口，例如 `18080`。 +- 已安排变更窗口，并确认短时间滚动重启 NodeLocal DNSCache Pod 可接受。 + +### 1.1 确认 NodeLocal DNSCache 资源名称 + +在目标集群中查找 NodeLocal DNSCache 的 DaemonSet 和 ConfigMap： + +```bash +kubectl get ds -A | grep -i node-local +kubectl get cm -A | grep -i node-local +``` + +记录实际的命名空间、DaemonSet 名称和 ConfigMap 名称。以下步骤使用默认名称作为示例： + +```bash +NS=kube-system +DS=node-local-dns +CM=node-local-dns +NEW_PORT=18080 +``` + +如果实际环境中的资源名称不同，请替换上述变量。 + +### 1.2 备份当前资源 + +备份修改前的 ConfigMap 和 DaemonSet，便于回滚： + +```bash +kubectl -n "$NS" get cm "$CM" -o yaml > node-local-dns-cm.backup.yaml +kubectl -n "$NS" get ds "$DS" -o yaml > node-local-dns-ds.backup.yaml +``` + +确认备份文件已生成： + +```bash +ls -l node-local-dns-cm.backup.yaml node-local-dns-ds.backup.yaml +``` + +### 1.3 修改 Corefile 健康检查端口 + +编辑 NodeLocal DNSCache ConfigMap： + +```bash +kubectl -n "$NS" edit cm "$CM" +``` + +将 Corefile 中的健康检查端口从 `8080` 改为新端口，例如： + +```text +health 127.0.0.1:8080 +``` + +改为： + +```text +health 127.0.0.1:18080 +``` + +不要修改 DNS 服务端口 `53`，也不要修改 metrics 端口 `9353`。 + +### 1.4 修改 DaemonSet 探针端口 + +编辑 NodeLocal DNSCache DaemonSet： + +```bash +kubectl -n "$NS" edit ds "$DS" +``` + +将 `node-cache` 容器的 `livenessProbe.httpGet.port` 从 `8080` 改为同一个新端口： + +```yaml +livenessProbe: + httpGet: + host: 127.0.0.1 + path: /health + port: 18080 +``` + +如果目标环境中的 DaemonSet 还包含 `readinessProbe`，并且该探针也访问 `/health` 或 `8080` 端口，需要同步改为同一个新端口。 + +### 1.5 等待 DaemonSet 滚动更新 + +修改 DaemonSet Pod 模板后，Kubernetes 会触发 DaemonSet 滚动更新。等待更新完成： + +```bash +kubectl -n "$NS" rollout status ds "$DS" +``` + +确认 NodeLocal DNSCache Pod 均处于运行状态： + +```bash +kubectl -n "$NS" get pods -l k8s-app="$DS" -o wide +``` + +如果环境中的 Pod 标签不是 `k8s-app=`，请根据实际 DaemonSet 的 `.spec.selector.matchLabels` 调整查询条件。 + +### 1.6 验证 + +确认 ConfigMap 已改为新端口： + +```bash +kubectl -n "$NS" get cm "$CM" -o yaml | grep 'health 127.0.0.1' +``` + +期望输出中的端口为新端口，例如： + +```text +health 127.0.0.1:18080 +``` + +确认 DaemonSet 探针端口已改为新端口： + +```bash +kubectl -n "$NS" get ds "$DS" -o yaml | grep -A5 -E 'livenessProbe|readinessProbe' +``` + +期望输出中的探针端口为新端口。 + +如需确认节点端口占用情况，可登录存在 NodeLocal DNSCache Pod 的节点执行： + +```bash +ss -ltnp | grep ':18080' +ss -ltnp | grep ':8080' +``` + +期望结果： + +- `18080` 端口由 NodeLocal DNSCache 进程监听。 +- `8080` 端口不再由 NodeLocal DNSCache 进程监听。 + +最后，从业务 Pod 中验证 DNS 解析正常： + +```bash +kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default.svc +``` + +期望可以正常解析 `kubernetes.default.svc`。 + +### 1.7 回滚 + +如果修改后 NodeLocal DNSCache Pod 异常，或 DNS 解析出现异常，使用备份文件恢复： + +```bash +kubectl apply -f node-local-dns-cm.backup.yaml +kubectl apply -f node-local-dns-ds.backup.yaml +kubectl -n "$NS" rollout status ds "$DS" +``` + +回滚完成后再次确认 Pod 状态和 DNS 解析： + +```bash +kubectl -n "$NS" get pods -l k8s-app="$DS" -o wide +kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default.svc +``` + +## 方案二：配置 CoreDNS ClusterIP 作为辅助 DNS server + +该方案适用于希望降低 NodeLocal DNSCache 单点风险的场景。配置后，新建 Pod 的 `/etc/resolv.conf` 中同时包含 NodeLocal DNSCache IP 和 CoreDNS ClusterIP。 + +该方案不能保证无感知故障切换。不同业务镜像中的 DNS 解析器重试行为不同；当第一个 DNS server 不可用时，部分工作负载可能需要等待超时后才尝试下一个 DNS server，故障期间 DNS 解析可能变慢。 + +操作前请确认： + +- 已获得目标集群的管理员 kubeconfig。 +- 已确认 NodeLocal DNSCache IP，例如 `169.254.20.10`。 +- 已确认 CoreDNS Service 的 ClusterIP。 +- 已安排变更窗口。修改 kubelet 配置需要重启 kubelet，且现有 Pod 的 `/etc/resolv.conf` 不会自动更新，需要重建受影响 Pod。 + +### 2.1 获取 CoreDNS ClusterIP + +查询 `kube-system` 命名空间中的 DNS Service： + +```bash +kubectl -n kube-system get svc kube-dns +``` + +记录 `CLUSTER-IP` 列的值。以下步骤使用 `10.96.0.10` 作为示例，请替换为实际值。 + +如果目标集群中的 DNS Service 不叫 `kube-dns`，先查找实际名称： + +```bash +kubectl -n kube-system get svc | grep -E 'kube-dns|coredns' +``` + +### 2.2 修改节点 kubelet 的 cluster-dns + +登录需要生效的节点，备份 kubelet 参数文件： + +```bash +sudo cp -a /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/kubeadm-flags.env.bak.$(date +%Y%m%d%H%M%S) +``` + +编辑 kubelet 参数文件： + +```bash +sudo vi /var/lib/kubelet/kubeadm-flags.env +``` + +将 kubelet 的 `--cluster-dns` 从单个 NodeLocal DNSCache IP 改为 NodeLocal DNSCache IP 和 CoreDNS ClusterIP 的组合。例如： + +```text +--cluster-dns=169.254.20.10 +``` + +改为： + +```text +--cluster-dns=169.254.20.10,10.96.0.10 +``` + +其中： + +- `169.254.20.10` 是 NodeLocal DNSCache IP。 +- `10.96.0.10` 是 CoreDNS ClusterIP。 + +不要删除同一行上的其他 kubelet 参数。 + +### 2.3 重启 kubelet + +保存配置后，重启 kubelet： + +```bash +sudo systemctl restart kubelet +``` + +如果目标操作系统不使用 systemd，请使用该环境支持的 kubelet 重启方式。 + +确认 kubelet 恢复运行： + +```bash +sudo systemctl status kubelet +``` + +### 2.4 重建受影响 Pod + +kubelet 的 `cluster-dns` 变更只影响新建 Pod。已有 Pod 的 `/etc/resolv.conf` 不会自动更新。 + +在变更窗口内，重建需要使用多 DNS server 的业务 Pod。重建方式取决于业务控制器类型，例如 Deployment 可以执行滚动重启： + +```bash +kubectl -n rollout restart deployment/ +kubectl -n rollout status deployment/ +``` + +### 2.5 验证 + +创建临时 Pod，确认 `/etc/resolv.conf` 中同时包含 NodeLocal DNSCache IP 和 CoreDNS ClusterIP： + +```bash +kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- cat /etc/resolv.conf +``` + +期望输出包含类似内容： + +```text +nameserver 169.254.20.10 +nameserver 10.96.0.10 +``` + +验证 DNS 解析正常： + +```bash +kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default.svc +``` + +如果集群启用了 NetworkPolicy，需要同时放行 Pod 访问 NodeLocal DNSCache IP 和 CoreDNS ClusterIP 的 TCP/UDP `53` 端口。 + +### 2.6 节点重建场景的持久化配置 + +如果集群通过重建节点方式升级，直接修改节点上的 `/var/lib/kubelet/kubeadm-flags.env` 会在节点重建后丢失。需要把同样的多地址 `cluster-dns` 值同步到集群模板中的每一处 `kubeletExtraArgs`： + +- `KubeadmControlPlane` → `initConfiguration` → `nodeRegistration` → `kubeletExtraArgs` +- `KubeadmControlPlane` → `joinConfiguration` → `nodeRegistration` → `kubeletExtraArgs` +- `KubeadmConfigTemplate` → `template` → `spec` → `joinConfiguration` → `nodeRegistration` → `kubeletExtraArgs` + +示例： + +```yaml +cluster-dns: "169.254.20.10,10.96.0.10" +``` + +### 2.7 回滚 + +如果配置多个 DNS server 后出现异常，登录已修改的节点，将 `/var/lib/kubelet/kubeadm-flags.env` 恢复为备份文件： + +```bash +sudo cp -a /var/lib/kubelet/kubeadm-flags.env.bak. /var/lib/kubelet/kubeadm-flags.env +sudo systemctl restart kubelet +``` + +然后重建受影响 Pod，使其 `/etc/resolv.conf` 重新生成。 + +## 长期建议 + +该问题的长期方案应在产品侧处理，例如： + +- 在 NodeLocal DNSCache 插件参数中暴露健康检查端口，并将该参数同时渲染到 Corefile 和 DaemonSet 探针。 +- 在 NodeLocal DNSCache 插件或集群配置中支持配置多个 kubelet `cluster-dns` 地址。 +- 在后续 DNS 架构演进中评估是否继续默认启用 NodeLocal DNSCache。 + +在产品化方案交付前，本文仅作为 S2 或实施人员的临时规避手册。 From 3a964d9c0bd5faee2265c6808becd79c41becb6f Mon Sep 17 00:00:00 2001 From: clyi Date: Wed, 12 Aug 2026 14:01:41 +0800 Subject: [PATCH 2/4] docs(acp): align nodelocaldns workarounds with kb style --- .../acp/NodeLocal_DNSCache_S2_Workarounds.md | 262 ++++------------- .../acp/NodeLocal_DNSCache_S2_Workarounds.md | 264 ++++-------------- 2 files changed, 119 insertions(+), 407 deletions(-) diff --git a/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md b/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md index ef116d3aa..0182463b8 100644 --- a/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md +++ b/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md @@ -7,23 +7,18 @@ ProductsVersion: - '4.2.x,4.3.x,4.4.x' --- -# S2 Temporary Workarounds for NodeLocal DNSCache Port 8080 Conflicts and DNS Single-Point Risk +# Temporary Workarounds for NodeLocal DNSCache Port 8080 Conflicts and DNS Single-Point Risk ## Problem -After the NodeLocal DNSCache plugin is installed in an ACP cluster, the following operational risks may occur: +After NodeLocal DNSCache is enabled in an ACP cluster, the following risks may occur: -- The `node-cache` Pod on each node runs with `hostNetwork: true` and listens on `127.0.0.1:8080` as its health check endpoint. If a business process, operations agent, or other node-level component also needs to use node-local port `8080`, a port conflict may occur. -- After NodeLocal DNSCache takes effect, newly created Pods use the node-local DNS address by default. If the `node-cache` Pod on a node is unavailable, DNS resolution for Pods on that node may fail. +- The `node-cache` Pod runs with `hostNetwork: true` and exposes its health check endpoint on the node loopback `127.0.0.1:8080`. If a business process, operations agent, or `hostNetwork` Pod on the same node also binds `127.0.0.1:8080` or `0.0.0.0:8080`, a port conflict occurs. +- After NodeLocal DNSCache takes effect, newly created Pods use the node-local DNS address as their DNS server. If the `node-cache` Pod on a node is unavailable, DNS resolution for Pods on that node may fail and does not automatically switch to CoreDNS without impact. ## Root Cause -The current NodeLocal DNSCache plugin does not expose the following settings in the installation parameters: - -- Health check port. -- Multiple DNS server configuration for kubelet `cluster-dns`. - -The generated Corefile and DaemonSet probe use `8080` by default: +The current NodeLocal DNSCache plugin exposes only the NodeLocal DNS IP parameter. It does not expose the health check port or multiple DNS server settings for kubelet. The generated Corefile and DaemonSet probe use `8080` by default: ```text health 127.0.0.1:8080 @@ -37,95 +32,50 @@ livenessProbe: port: 8080 ``` -The plugin installation job also configures kubelet `--cluster-dns` to the NodeLocal DNSCache IP. If CoreDNS ClusterIP must be used as an additional DNS server, S2 or implementation engineers need to temporarily modify the kubelet configuration on the target cluster. - -## Temporary Workarounds - -This article provides two independent S2 temporary workarounds: +The plugin installation job also configures kubelet `--cluster-dns` to the NodeLocal DNSCache IP. To release port `8080`, or to use CoreDNS ClusterIP as an additional DNS server, you need to temporarily modify runtime resources or node kubelet configuration. -- Workaround 1: Change the NodeLocal DNSCache health check port to avoid node port `8080` conflicts. -- Workaround 2: Configure multiple DNS servers and use CoreDNS ClusterIP as an additional DNS server to reduce the impact when node-local DNS is unavailable. +## Resolution -These workarounds are not persistent product capabilities. Manual changes may be overwritten after plugin upgrade, plugin reinstall, platform reconciliation, chart re-rendering, or node rebuild. For long-term use, the required configuration should be productized. +The following workarounds are temporary. Manual changes may be overwritten after plugin upgrade, plugin reinstall, platform reconciliation, chart re-rendering, or node rebuild. Perform the change in a maintenance window and keep backups before editing resources. -## Workaround 1: Avoid NodeLocal DNSCache Port 8080 Conflicts +### Workaround 1: Change the NodeLocal DNSCache health check port -Use this workaround when a business process, operations agent, or other node-level component must use node-local port `8080`. +Use this workaround when node-local port `8080` must be released. Change both the Corefile `health` port and the DaemonSet probe port. The two values must stay consistent. -Before you start, confirm that: - -- You have an administrator kubeconfig for the target cluster. -- The business side really needs to release node-local port `8080`. -- You have selected a new port that is not used by other node components, for example `18080`. -- You have scheduled a change window, and a short rolling restart of NodeLocal DNSCache Pods is acceptable. - -### 1.1 Confirm NodeLocal DNSCache Resource Names - -Find the NodeLocal DNSCache DaemonSet and ConfigMap in the target cluster: - -```bash -kubectl get ds -A | grep -i node-local -kubectl get cm -A | grep -i node-local -``` - -Record the actual namespace, DaemonSet name, and ConfigMap name. The following steps use the default names as examples: +Confirm resource names and back up current resources: ```bash NS=kube-system DS=node-local-dns CM=node-local-dns -NEW_PORT=18080 -``` - -If the resource names are different in the actual environment, replace these variables. -### 1.2 Back Up Current Resources - -Back up the ConfigMap and DaemonSet before making changes: - -```bash kubectl -n "$NS" get cm "$CM" -o yaml > node-local-dns-cm.backup.yaml kubectl -n "$NS" get ds "$DS" -o yaml > node-local-dns-ds.backup.yaml ``` -Confirm that the backup files are generated: +If the DaemonSet or ConfigMap uses a different name in the actual environment, find the resource first: ```bash -ls -l node-local-dns-cm.backup.yaml node-local-dns-ds.backup.yaml +kubectl get ds -A | grep -i node-local +kubectl get cm -A | grep -i node-local ``` -### 1.3 Change the Corefile Health Check Port - -Edit the NodeLocal DNSCache ConfigMap: +Edit the ConfigMap and change the health check port in the Corefile to an unused port, for example `18080`: ```bash kubectl -n "$NS" edit cm "$CM" ``` -Change the health check port in the Corefile from `8080` to the new port. For example, change: - -```text -health 127.0.0.1:8080 -``` - -to: - ```text health 127.0.0.1:18080 ``` -Do not change the DNS service port `53` or the metrics port `9353`. - -### 1.4 Change the DaemonSet Probe Port - -Edit the NodeLocal DNSCache DaemonSet: +Edit the DaemonSet and change `livenessProbe.httpGet.port` of the `node-cache` container to the same port: ```bash kubectl -n "$NS" edit ds "$DS" ``` -Change `livenessProbe.httpGet.port` of the `node-cache` container from `8080` to the same new port: - ```yaml livenessProbe: httpGet: @@ -134,174 +84,70 @@ livenessProbe: port: 18080 ``` -If the DaemonSet in the target environment also includes a `readinessProbe`, and that probe also accesses `/health` or port `8080`, change it to the same new port. +If the target environment also has a `readinessProbe` that accesses `/health` or `8080`, change it as well. Do not change DNS service port `53` or metrics port `9353`. -### 1.5 Wait for the DaemonSet Rolling Update - -After the DaemonSet Pod template is changed, Kubernetes starts a DaemonSet rolling update. Wait for the update to complete: +Wait for the DaemonSet rolling update to complete and verify DNS resolution: ```bash kubectl -n "$NS" rollout status ds "$DS" +kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default.svc ``` -Confirm that all NodeLocal DNSCache Pods are running: - -```bash -kubectl -n "$NS" get pods -l k8s-app="$DS" -o wide -``` - -If the Pod label in the environment is not `k8s-app=`, adjust the query label according to `.spec.selector.matchLabels` of the DaemonSet. - -### 1.6 Verify the Change - -Confirm that the ConfigMap uses the new port: - -```bash -kubectl -n "$NS" get cm "$CM" -o yaml | grep 'health 127.0.0.1' -``` - -The expected output should contain the new port, for example: - -```text -health 127.0.0.1:18080 -``` - -Confirm that the DaemonSet probe port uses the new port: - -```bash -kubectl -n "$NS" get ds "$DS" -o yaml | grep -A5 -E 'livenessProbe|readinessProbe' -``` - -The expected output should show the new probe port. - -If you need to check node port usage, log in to a node that has a NodeLocal DNSCache Pod and run: +To confirm node port listeners, log in to a node running the `node-cache` Pod and run: ```bash ss -ltnp | grep ':18080' ss -ltnp | grep ':8080' ``` -Expected results: - -- Port `18080` is listened on by the NodeLocal DNSCache process. -- Port `8080` is no longer listened on by the NodeLocal DNSCache process. - -Finally, verify DNS resolution from a business Pod or a temporary Pod: - -```bash -kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default.svc -``` - -The command should resolve `kubernetes.default.svc` successfully. - -### 1.7 Roll Back - -If the NodeLocal DNSCache Pod becomes abnormal or DNS resolution fails after the change, restore the backup files: - -```bash -kubectl apply -f node-local-dns-cm.backup.yaml -kubectl apply -f node-local-dns-ds.backup.yaml -kubectl -n "$NS" rollout status ds "$DS" -``` - -After rollback, verify Pod status and DNS resolution again: +The expected result is that `18080` is listened on by NodeLocal DNSCache, and `8080` is no longer listened on by NodeLocal DNSCache. -```bash -kubectl -n "$NS" get pods -l k8s-app="$DS" -o wide -kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default.svc -``` - -## Workaround 2: Configure CoreDNS ClusterIP as an Additional DNS Server +### Workaround 2: Configure CoreDNS ClusterIP as an additional DNS server Use this workaround when you want to reduce the single-point impact of NodeLocal DNSCache. After configuration, newly created Pods have both the NodeLocal DNSCache IP and CoreDNS ClusterIP in `/etc/resolv.conf`. -This workaround does not guarantee transparent failover. DNS resolver retry behavior differs between business images. When the first DNS server is unavailable, some workloads may wait for timeout before trying the next DNS server, which can slow down DNS resolution during the failure. - -Before you start, confirm that: - -- You have an administrator kubeconfig for the target cluster. -- You have confirmed the NodeLocal DNSCache IP, for example `169.254.20.10`. -- You have confirmed the CoreDNS Service ClusterIP. -- You have scheduled a change window. Changing kubelet configuration requires restarting kubelet, and existing Pods do not automatically update `/etc/resolv.conf`; affected Pods need to be recreated. - -### 2.1 Get CoreDNS ClusterIP +This is not a transparent failover mechanism. DNS resolver retry behavior differs between business images. When the first DNS server is unavailable, some workloads may wait for timeout before trying the next DNS server, which can slow down DNS resolution during the failure. -Query the DNS Service in the `kube-system` namespace: +Get the CoreDNS ClusterIP: ```bash kubectl -n kube-system get svc kube-dns ``` -Record the value in the `CLUSTER-IP` column. The following steps use `10.96.0.10` as an example; replace it with the actual value. - If the DNS Service in the target cluster is not named `kube-dns`, find the actual name first: ```bash kubectl -n kube-system get svc | grep -E 'kube-dns|coredns' ``` -### 2.2 Change kubelet cluster-dns on Nodes - -Log in to each node that needs the change, and back up the kubelet argument file: +Log in to each node that needs the change, then back up and edit the kubelet argument file: ```bash sudo cp -a /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/kubeadm-flags.env.bak.$(date +%Y%m%d%H%M%S) -``` - -Edit the kubelet argument file: - -```bash sudo vi /var/lib/kubelet/kubeadm-flags.env ``` -Change kubelet `--cluster-dns` from the single NodeLocal DNSCache IP to a combination of NodeLocal DNSCache IP and CoreDNS ClusterIP. For example, change: - -```text ---cluster-dns=169.254.20.10 -``` - -to: +Change kubelet `--cluster-dns` from a single NodeLocal DNSCache IP to a combination of NodeLocal DNSCache IP and CoreDNS ClusterIP. For example: ```text --cluster-dns=169.254.20.10,10.96.0.10 ``` -Where: - -- `169.254.20.10` is the NodeLocal DNSCache IP. -- `10.96.0.10` is the CoreDNS ClusterIP. - -Do not remove other kubelet arguments on the same line. +In this example, `169.254.20.10` is the NodeLocal DNSCache IP and `10.96.0.10` is the CoreDNS ClusterIP. Do not remove other kubelet arguments on the same line. -### 2.3 Restart kubelet - -After saving the configuration, restart kubelet: +Restart kubelet after saving the change: ```bash sudo systemctl restart kubelet ``` -If the target operating system does not use systemd, use the kubelet restart method supported by that environment. - -Confirm that kubelet is running again: - -```bash -sudo systemctl status kubelet -``` - -### 2.4 Recreate Affected Pods - -The kubelet `cluster-dns` change only affects newly created Pods. Existing Pods do not automatically update `/etc/resolv.conf`. - -During the change window, recreate the business Pods that need to use multiple DNS servers. The recreation method depends on the workload controller type. For example, for a Deployment: +The kubelet `cluster-dns` change only affects newly created Pods. Existing Pods do not automatically update `/etc/resolv.conf`. Recreate the affected business Pods during the maintenance window. For example: ```bash kubectl -n rollout restart deployment/ kubectl -n rollout status deployment/ ``` -### 2.5 Verify the Change - Create a temporary Pod and confirm that `/etc/resolv.conf` contains both the NodeLocal DNSCache IP and CoreDNS ClusterIP: ```bash @@ -315,31 +161,41 @@ nameserver 169.254.20.10 nameserver 10.96.0.10 ``` -Verify DNS resolution: +If NetworkPolicy is enabled in the cluster, allow Pods to access both the NodeLocal DNSCache IP and CoreDNS ClusterIP on TCP/UDP port `53`. + +## Diagnostic Steps + +Confirm that NodeLocal DNSCache is installed and Pods are Ready: ```bash -kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default.svc +kubectl -n kube-system get pods -l k8s-app=node-local-dns -o wide +kubectl -n kube-system rollout status ds/node-local-dns ``` -If NetworkPolicy is enabled in the cluster, allow Pods to access both the NodeLocal DNSCache IP and CoreDNS ClusterIP on TCP/UDP port `53`. +Confirm whether the current Corefile and DaemonSet probe still use `8080`: -### 2.6 Persist the Configuration for Node Rebuild Scenarios +```bash +kubectl -n kube-system get cm node-local-dns -o yaml | grep 'health 127.0.0.1' +kubectl -n kube-system get ds node-local-dns -o yaml | grep -A5 -E 'livenessProbe|readinessProbe' +``` -If the cluster is upgraded by rebuilding nodes, directly changing `/var/lib/kubelet/kubeadm-flags.env` on nodes is lost after node rebuild. You need to synchronize the same multi-address `cluster-dns` value to every `kubeletExtraArgs` location in the cluster template: +Confirm the DNS servers used by newly created Pods: -- `KubeadmControlPlane` → `initConfiguration` → `nodeRegistration` → `kubeletExtraArgs` -- `KubeadmControlPlane` → `joinConfiguration` → `nodeRegistration` → `kubeletExtraArgs` -- `KubeadmConfigTemplate` → `template` → `spec` → `joinConfiguration` → `nodeRegistration` → `kubeletExtraArgs` +```bash +kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- cat /etc/resolv.conf +``` -Example: +## Rollback -```yaml -cluster-dns: "169.254.20.10,10.96.0.10" -``` +If changing the health check port causes problems, restore the backed-up ConfigMap and DaemonSet: -### 2.7 Roll Back +```bash +kubectl apply -f node-local-dns-cm.backup.yaml +kubectl apply -f node-local-dns-ds.backup.yaml +kubectl -n kube-system rollout status ds/node-local-dns +``` -If problems occur after configuring multiple DNS servers, log in to the modified nodes and restore `/var/lib/kubelet/kubeadm-flags.env` from the backup file: +If configuring multiple DNS servers causes problems, log in to the modified nodes, restore `/var/lib/kubelet/kubeadm-flags.env` from the backup, and restart kubelet: ```bash sudo cp -a /var/lib/kubelet/kubeadm-flags.env.bak. /var/lib/kubelet/kubeadm-flags.env @@ -348,12 +204,12 @@ sudo systemctl restart kubelet Then recreate the affected Pods so their `/etc/resolv.conf` is regenerated. -## Long-Term Recommendation +## Related Information -The long-term fix should be implemented on the product side. For example: +If the cluster is upgraded by rebuilding nodes, directly changing `/var/lib/kubelet/kubeadm-flags.env` on nodes is lost after node rebuild. You need to synchronize the same multi-address `cluster-dns` value to every `kubeletExtraArgs` location in the cluster template: -- Expose the health check port in the NodeLocal DNSCache plugin parameters, and render the value to both the Corefile and DaemonSet probe. -- Support multiple kubelet `cluster-dns` addresses in the NodeLocal DNSCache plugin or cluster configuration. -- Evaluate whether NodeLocal DNSCache should continue to be enabled by default in future DNS architecture evolution. +- `KubeadmControlPlane` → `initConfiguration` → `nodeRegistration` → `kubeletExtraArgs` +- `KubeadmControlPlane` → `joinConfiguration` → `nodeRegistration` → `kubeletExtraArgs` +- `KubeadmConfigTemplate` → `template` → `spec` → `joinConfiguration` → `nodeRegistration` → `kubeletExtraArgs` -Before the productized solution is delivered, this article should only be used as a temporary workaround guide for S2 or implementation engineers. +The long-term fix should be implemented on the product side, for example by exposing the health check port in the NodeLocal DNSCache plugin parameters, or by supporting multiple kubelet `cluster-dns` addresses in the plugin or cluster configuration. diff --git a/docs/zh/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md b/docs/zh/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md index b7693949c..5510da5ed 100644 --- a/docs/zh/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md +++ b/docs/zh/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md @@ -7,23 +7,18 @@ ProductsVersion: - '4.2.x,4.3.x,4.4.x' --- -# NodeLocal DNSCache 8080 端口冲突和 DNS 单点风险的 S2 临时规避方案 +# NodeLocal DNSCache 8080 端口冲突和 DNS 单点风险的临时规避方案 ## 问题 -在 ACP 集群中安装 NodeLocal DNSCache 插件后，可能遇到以下两个运维风险： +在 ACP 集群中启用 NodeLocal DNSCache 后，可能出现以下风险： -- 每个节点上的 `node-cache` Pod 使用 `hostNetwork: true` 运行，并在节点本地监听 `127.0.0.1:8080` 作为健康检查端口。如果业务进程、运维代理或其他节点级组件也需要使用节点本地 `8080` 端口，可能出现端口冲突。 -- NodeLocal DNSCache 生效后，新建 Pod 默认使用节点本地 DNS 地址。当某个节点上的 `node-cache` Pod 不可用时，该节点上 Pod 的 DNS 解析可能失败。 +- `node-cache` Pod 使用 `hostNetwork: true`，并在节点 loopback `127.0.0.1:8080` 上暴露健康检查端点。如果节点上的业务进程、运维代理或 `hostNetwork` Pod 也绑定 `127.0.0.1:8080` 或 `0.0.0.0:8080`，会发生端口冲突。 +- NodeLocal DNSCache 生效后，新建 Pod 使用节点本地 DNS 地址作为 DNS 服务器。当某个节点上的 `node-cache` Pod 不可用时，该节点上 Pod 的 DNS 解析可能失败，且不会自动无感切换到 CoreDNS。 ## 根本原因 -当前 NodeLocal DNSCache 插件未在安装参数中暴露以下配置： - -- 健康检查端口。 -- kubelet `cluster-dns` 的多 DNS server 配置。 - -插件生成的资源中，Corefile 和 DaemonSet 探针默认使用 `8080`： +当前 NodeLocal DNSCache 插件只暴露 NodeLocal DNS IP 参数，未暴露健康检查端口或 kubelet 多 DNS server 参数。插件生成的 Corefile 和 DaemonSet 探针默认使用 `8080`： ```text health 127.0.0.1:8080 @@ -37,95 +32,50 @@ livenessProbe: port: 8080 ``` -同时，插件安装任务会把 kubelet `--cluster-dns` 配置为 NodeLocal DNSCache IP。若需要把 CoreDNS ClusterIP 作为辅助 DNS server，需要由 S2 或实施人员在目标集群中临时修改 kubelet 配置。 - -## 临时方案 - -本文包含两个可独立执行的 S2 临时方案： +插件安装任务还会把 kubelet `--cluster-dns` 配置为 NodeLocal DNSCache IP。如果需要释放 `8080`，或把 CoreDNS ClusterIP 作为辅助 DNS server，需要临时修改运行中资源或节点 kubelet 配置。 -- 方案一：修改 NodeLocal DNSCache 健康检查端口，规避节点 `8080` 端口冲突。 -- 方案二：配置多个 DNS server，将 CoreDNS ClusterIP 作为辅助 DNS，降低节点本地 DNS 不可用时的影响。 +## 解决方案 -这些方案不是持久化产品能力。插件升级、重装、平台调谐、重新渲染 chart 或节点重建后，手工修改可能被覆盖。如需长期使用，应推动产品化支持。 +以下方案仅用于临时规避。插件升级、重装、平台调谐、chart 重新渲染或节点重建后，手工修改可能被覆盖。建议在变更窗口执行，并在修改前保留备份。 -## 方案一：规避 NodeLocal DNSCache 占用节点 8080 端口 +### 方案一：修改 NodeLocal DNSCache 健康检查端口 -该方案适用于业务进程、运维代理或其他节点级组件必须使用节点本地 `8080` 端口的场景。 +该方案适用于必须释放节点本地 `8080` 端口的场景。需要同时修改 Corefile 中的 `health` 端口和 DaemonSet 探针端口，两个位置必须保持一致。 -操作前请确认： - -- 已获得目标集群的管理员 kubeconfig。 -- 已确认业务侧确实需要释放节点本地 `8080` 端口。 -- 已选择一个未被节点上其他组件占用的新端口，例如 `18080`。 -- 已安排变更窗口，并确认短时间滚动重启 NodeLocal DNSCache Pod 可接受。 - -### 1.1 确认 NodeLocal DNSCache 资源名称 - -在目标集群中查找 NodeLocal DNSCache 的 DaemonSet 和 ConfigMap： - -```bash -kubectl get ds -A | grep -i node-local -kubectl get cm -A | grep -i node-local -``` - -记录实际的命名空间、DaemonSet 名称和 ConfigMap 名称。以下步骤使用默认名称作为示例： +先确认资源名称，并备份当前资源： ```bash NS=kube-system DS=node-local-dns CM=node-local-dns -NEW_PORT=18080 -``` - -如果实际环境中的资源名称不同，请替换上述变量。 -### 1.2 备份当前资源 - -备份修改前的 ConfigMap 和 DaemonSet，便于回滚： - -```bash kubectl -n "$NS" get cm "$CM" -o yaml > node-local-dns-cm.backup.yaml kubectl -n "$NS" get ds "$DS" -o yaml > node-local-dns-ds.backup.yaml ``` -确认备份文件已生成： +如果实际环境中的 DaemonSet 或 ConfigMap 名称不同，可通过以下命令查找： ```bash -ls -l node-local-dns-cm.backup.yaml node-local-dns-ds.backup.yaml +kubectl get ds -A | grep -i node-local +kubectl get cm -A | grep -i node-local ``` -### 1.3 修改 Corefile 健康检查端口 - -编辑 NodeLocal DNSCache ConfigMap： +编辑 ConfigMap，把 Corefile 中的健康检查端口改为未被占用的新端口，例如 `18080`： ```bash kubectl -n "$NS" edit cm "$CM" ``` -将 Corefile 中的健康检查端口从 `8080` 改为新端口，例如： - -```text -health 127.0.0.1:8080 -``` - -改为： - ```text health 127.0.0.1:18080 ``` -不要修改 DNS 服务端口 `53`，也不要修改 metrics 端口 `9353`。 - -### 1.4 修改 DaemonSet 探针端口 - -编辑 NodeLocal DNSCache DaemonSet： +编辑 DaemonSet，把 `node-cache` 容器的 `livenessProbe.httpGet.port` 改为同一个端口： ```bash kubectl -n "$NS" edit ds "$DS" ``` -将 `node-cache` 容器的 `livenessProbe.httpGet.port` 从 `8080` 改为同一个新端口： - ```yaml livenessProbe: httpGet: @@ -134,212 +84,118 @@ livenessProbe: port: 18080 ``` -如果目标环境中的 DaemonSet 还包含 `readinessProbe`，并且该探针也访问 `/health` 或 `8080` 端口，需要同步改为同一个新端口。 +如果目标环境中还存在访问 `/health` 或 `8080` 的 `readinessProbe`，也需要同步修改。不要修改 DNS 服务端口 `53` 或 metrics 端口 `9353`。 -### 1.5 等待 DaemonSet 滚动更新 - -修改 DaemonSet Pod 模板后，Kubernetes 会触发 DaemonSet 滚动更新。等待更新完成： +等待 DaemonSet 滚动更新完成，并验证 DNS 解析正常： ```bash kubectl -n "$NS" rollout status ds "$DS" +kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default.svc ``` -确认 NodeLocal DNSCache Pod 均处于运行状态： - -```bash -kubectl -n "$NS" get pods -l k8s-app="$DS" -o wide -``` - -如果环境中的 Pod 标签不是 `k8s-app=`，请根据实际 DaemonSet 的 `.spec.selector.matchLabels` 调整查询条件。 - -### 1.6 验证 - -确认 ConfigMap 已改为新端口： - -```bash -kubectl -n "$NS" get cm "$CM" -o yaml | grep 'health 127.0.0.1' -``` - -期望输出中的端口为新端口，例如： - -```text -health 127.0.0.1:18080 -``` - -确认 DaemonSet 探针端口已改为新端口： - -```bash -kubectl -n "$NS" get ds "$DS" -o yaml | grep -A5 -E 'livenessProbe|readinessProbe' -``` - -期望输出中的探针端口为新端口。 - -如需确认节点端口占用情况，可登录存在 NodeLocal DNSCache Pod 的节点执行： +如需确认节点端口监听，可登录运行 `node-cache` Pod 的节点执行： ```bash ss -ltnp | grep ':18080' ss -ltnp | grep ':8080' ``` -期望结果： - -- `18080` 端口由 NodeLocal DNSCache 进程监听。 -- `8080` 端口不再由 NodeLocal DNSCache 进程监听。 - -最后，从业务 Pod 中验证 DNS 解析正常： - -```bash -kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default.svc -``` - -期望可以正常解析 `kubernetes.default.svc`。 - -### 1.7 回滚 - -如果修改后 NodeLocal DNSCache Pod 异常，或 DNS 解析出现异常，使用备份文件恢复： - -```bash -kubectl apply -f node-local-dns-cm.backup.yaml -kubectl apply -f node-local-dns-ds.backup.yaml -kubectl -n "$NS" rollout status ds "$DS" -``` - -回滚完成后再次确认 Pod 状态和 DNS 解析： - -```bash -kubectl -n "$NS" get pods -l k8s-app="$DS" -o wide -kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default.svc -``` - -## 方案二：配置 CoreDNS ClusterIP 作为辅助 DNS server - -该方案适用于希望降低 NodeLocal DNSCache 单点风险的场景。配置后，新建 Pod 的 `/etc/resolv.conf` 中同时包含 NodeLocal DNSCache IP 和 CoreDNS ClusterIP。 - -该方案不能保证无感知故障切换。不同业务镜像中的 DNS 解析器重试行为不同；当第一个 DNS server 不可用时，部分工作负载可能需要等待超时后才尝试下一个 DNS server，故障期间 DNS 解析可能变慢。 +预期 `18080` 由 NodeLocal DNSCache 监听，`8080` 不再由 NodeLocal DNSCache 监听。 -操作前请确认： +### 方案二：配置 CoreDNS ClusterIP 作为辅助 DNS server -- 已获得目标集群的管理员 kubeconfig。 -- 已确认 NodeLocal DNSCache IP，例如 `169.254.20.10`。 -- 已确认 CoreDNS Service 的 ClusterIP。 -- 已安排变更窗口。修改 kubelet 配置需要重启 kubelet，且现有 Pod 的 `/etc/resolv.conf` 不会自动更新，需要重建受影响 Pod。 +该方案适用于希望降低 NodeLocal DNSCache 单点影响的场景。配置后，新建 Pod 的 `/etc/resolv.conf` 中同时包含 NodeLocal DNSCache IP 和 CoreDNS ClusterIP。 -### 2.1 获取 CoreDNS ClusterIP +该方案不是无感故障切换机制。不同业务镜像中的 DNS 解析器重试行为不同；当第一个 DNS server 不可用时，部分工作负载可能需要等待超时后才尝试下一个 DNS server，故障期间 DNS 解析可能变慢。 -查询 `kube-system` 命名空间中的 DNS Service： +先查询 CoreDNS ClusterIP： ```bash kubectl -n kube-system get svc kube-dns ``` -记录 `CLUSTER-IP` 列的值。以下步骤使用 `10.96.0.10` 作为示例，请替换为实际值。 - 如果目标集群中的 DNS Service 不叫 `kube-dns`，先查找实际名称： ```bash kubectl -n kube-system get svc | grep -E 'kube-dns|coredns' ``` -### 2.2 修改节点 kubelet 的 cluster-dns - -登录需要生效的节点，备份 kubelet 参数文件： +登录需要生效的节点，备份并编辑 kubelet 参数文件： ```bash sudo cp -a /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/kubeadm-flags.env.bak.$(date +%Y%m%d%H%M%S) -``` - -编辑 kubelet 参数文件： - -```bash sudo vi /var/lib/kubelet/kubeadm-flags.env ``` 将 kubelet 的 `--cluster-dns` 从单个 NodeLocal DNSCache IP 改为 NodeLocal DNSCache IP 和 CoreDNS ClusterIP 的组合。例如： -```text ---cluster-dns=169.254.20.10 -``` - -改为： - ```text --cluster-dns=169.254.20.10,10.96.0.10 ``` -其中： - -- `169.254.20.10` 是 NodeLocal DNSCache IP。 -- `10.96.0.10` 是 CoreDNS ClusterIP。 - -不要删除同一行上的其他 kubelet 参数。 +其中 `169.254.20.10` 是 NodeLocal DNSCache IP，`10.96.0.10` 是 CoreDNS ClusterIP。不要删除同一行上的其他 kubelet 参数。 -### 2.3 重启 kubelet - -保存配置后，重启 kubelet： +保存后重启 kubelet： ```bash sudo systemctl restart kubelet ``` -如果目标操作系统不使用 systemd，请使用该环境支持的 kubelet 重启方式。 - -确认 kubelet 恢复运行： - -```bash -sudo systemctl status kubelet -``` - -### 2.4 重建受影响 Pod - -kubelet 的 `cluster-dns` 变更只影响新建 Pod。已有 Pod 的 `/etc/resolv.conf` 不会自动更新。 - -在变更窗口内，重建需要使用多 DNS server 的业务 Pod。重建方式取决于业务控制器类型，例如 Deployment 可以执行滚动重启： +kubelet 的 `cluster-dns` 变更只影响新建 Pod，已有 Pod 的 `/etc/resolv.conf` 不会自动更新。需要在变更窗口内重建受影响业务 Pod，例如： ```bash kubectl -n rollout restart deployment/ kubectl -n rollout status deployment/ ``` -### 2.5 验证 - 创建临时 Pod，确认 `/etc/resolv.conf` 中同时包含 NodeLocal DNSCache IP 和 CoreDNS ClusterIP： ```bash kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- cat /etc/resolv.conf ``` -期望输出包含类似内容： +预期输出包含类似内容： ```text nameserver 169.254.20.10 nameserver 10.96.0.10 ``` -验证 DNS 解析正常： +如果集群启用了 NetworkPolicy，需要同时放行 Pod 访问 NodeLocal DNSCache IP 和 CoreDNS ClusterIP 的 TCP/UDP `53` 端口。 + +## 诊断步骤 + +确认 NodeLocal DNSCache 是否安装、Pod 是否就绪： ```bash -kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default.svc +kubectl -n kube-system get pods -l k8s-app=node-local-dns -o wide +kubectl -n kube-system rollout status ds/node-local-dns ``` -如果集群启用了 NetworkPolicy，需要同时放行 Pod 访问 NodeLocal DNSCache IP 和 CoreDNS ClusterIP 的 TCP/UDP `53` 端口。 +确认当前 Corefile 和 DaemonSet 探针是否仍使用 `8080`： -### 2.6 节点重建场景的持久化配置 +```bash +kubectl -n kube-system get cm node-local-dns -o yaml | grep 'health 127.0.0.1' +kubectl -n kube-system get ds node-local-dns -o yaml | grep -A5 -E 'livenessProbe|readinessProbe' +``` -如果集群通过重建节点方式升级，直接修改节点上的 `/var/lib/kubelet/kubeadm-flags.env` 会在节点重建后丢失。需要把同样的多地址 `cluster-dns` 值同步到集群模板中的每一处 `kubeletExtraArgs`： +确认新建 Pod 实际使用的 DNS server： -- `KubeadmControlPlane` → `initConfiguration` → `nodeRegistration` → `kubeletExtraArgs` -- `KubeadmControlPlane` → `joinConfiguration` → `nodeRegistration` → `kubeletExtraArgs` -- `KubeadmConfigTemplate` → `template` → `spec` → `joinConfiguration` → `nodeRegistration` → `kubeletExtraArgs` +```bash +kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- cat /etc/resolv.conf +``` -示例： +## 回滚 -```yaml -cluster-dns: "169.254.20.10,10.96.0.10" -``` +如果修改健康检查端口后异常，恢复备份的 ConfigMap 和 DaemonSet： -### 2.7 回滚 +```bash +kubectl apply -f node-local-dns-cm.backup.yaml +kubectl apply -f node-local-dns-ds.backup.yaml +kubectl -n kube-system rollout status ds/node-local-dns +``` -如果配置多个 DNS server 后出现异常，登录已修改的节点，将 `/var/lib/kubelet/kubeadm-flags.env` 恢复为备份文件： +如果配置多个 DNS server 后异常，登录已修改节点，将 `/var/lib/kubelet/kubeadm-flags.env` 恢复为备份文件并重启 kubelet： ```bash sudo cp -a /var/lib/kubelet/kubeadm-flags.env.bak. /var/lib/kubelet/kubeadm-flags.env @@ -348,12 +204,12 @@ sudo systemctl restart kubelet 然后重建受影响 Pod，使其 `/etc/resolv.conf` 重新生成。 -## 长期建议 +## 相关说明 -该问题的长期方案应在产品侧处理，例如： +如果集群通过重建节点方式升级，直接修改节点上的 `/var/lib/kubelet/kubeadm-flags.env` 会在节点重建后丢失。需要把同样的多地址 `cluster-dns` 值同步到集群模板中的每一处 `kubeletExtraArgs`： -- 在 NodeLocal DNSCache 插件参数中暴露健康检查端口，并将该参数同时渲染到 Corefile 和 DaemonSet 探针。 -- 在 NodeLocal DNSCache 插件或集群配置中支持配置多个 kubelet `cluster-dns` 地址。 -- 在后续 DNS 架构演进中评估是否继续默认启用 NodeLocal DNSCache。 +- `KubeadmControlPlane` → `initConfiguration` → `nodeRegistration` → `kubeletExtraArgs` +- `KubeadmControlPlane` → `joinConfiguration` → `nodeRegistration` → `kubeletExtraArgs` +- `KubeadmConfigTemplate` → `template` → `spec` → `joinConfiguration` → `nodeRegistration` → `kubeletExtraArgs` -在产品化方案交付前，本文仅作为 S2 或实施人员的临时规避手册。 +长期方案应在产品侧处理，例如在 NodeLocal DNSCache 插件参数中暴露健康检查端口，或在插件/集群配置中支持多个 kubelet `cluster-dns` 地址。 From 2b5ddd605466680af032f8f24b15f85ce201f7ad Mon Sep 17 00:00:00 2001 From: clyi Date: Wed, 12 Aug 2026 14:31:23 +0800 Subject: [PATCH 3/4] docs(acp): add nodelocaldns metrics workaround --- .../acp/NodeLocal_DNSCache_S2_Workarounds.md | 68 ++++++++++++++++++- .../acp/NodeLocal_DNSCache_S2_Workarounds.md | 68 ++++++++++++++++++- 2 files changed, 132 insertions(+), 4 deletions(-) diff --git a/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md b/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md index 0182463b8..f7ec5dcfd 100644 --- a/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md +++ b/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md @@ -7,7 +7,7 @@ ProductsVersion: - '4.2.x,4.3.x,4.4.x' --- -# Temporary Workarounds for NodeLocal DNSCache Port 8080 Conflicts and DNS Single-Point Risk +# Temporary Workarounds for NodeLocal DNSCache Port Conflicts, DNS Single-Point Risk, and Metrics Access Issues ## Problem @@ -15,6 +15,7 @@ After NodeLocal DNSCache is enabled in an ACP cluster, the following risks may o - The `node-cache` Pod runs with `hostNetwork: true` and exposes its health check endpoint on the node loopback `127.0.0.1:8080`. If a business process, operations agent, or `hostNetwork` Pod on the same node also binds `127.0.0.1:8080` or `0.0.0.0:8080`, a port conflict occurs. - After NodeLocal DNSCache takes effect, newly created Pods use the node-local DNS address as their DNS server. If the `node-cache` Pod on a node is unavailable, DNS resolution for Pods on that node may fail and does not automatically switch to CoreDNS without impact. +- If the Corefile `prometheus` directive binds to the NodeLocal DNSCache IP, for example `169.254.20.10:9253`, external monitoring systems or dashboards may fail to access the metrics endpoint. ## Root Cause @@ -34,6 +35,8 @@ livenessProbe: The plugin installation job also configures kubelet `--cluster-dns` to the NodeLocal DNSCache IP. To release port `8080`, or to use CoreDNS ClusterIP as an additional DNS server, you need to temporarily modify runtime resources or node kubelet configuration. +For metrics access issues, the common root cause is that the Corefile `prometheus` directive binds to the NodeLocal DNSCache IP instead of listening only on a port. If the external monitoring path cannot reach that node-local address, metrics cannot be collected. + ## Resolution The following workarounds are temporary. Manual changes may be overwritten after plugin upgrade, plugin reinstall, platform reconciliation, chart re-rendering, or node rebuild. Perform the change in a maintenance window and keep backups before editing resources. @@ -163,6 +166,53 @@ nameserver 10.96.0.10 If NetworkPolicy is enabled in the cluster, allow Pods to access both the NodeLocal DNSCache IP and CoreDNS ClusterIP on TCP/UDP port `53`. +### Workaround 3: Remove the fixed IP binding from Prometheus metrics + +Use this workaround when external monitoring systems or dashboards cannot access NodeLocal DNSCache metrics. Change only the `prometheus` listen address in the Corefile. Do not change the DNS service port. + +Back up the current ConfigMap first: + +```bash +NS=kube-system +CM=node-local-dns +DS=node-local-dns + +kubectl -n "$NS" get cm "$CM" -o yaml > node-local-dns-cm.backup.yaml +``` + +Edit the ConfigMap: + +```bash +kubectl -n "$NS" edit cm "$CM" +``` + +Change the `prometheus` directive that binds to a fixed IP: + +```text +prometheus 169.254.20.10:9253 +``` + +to listen only on the port: + +```text +prometheus :9253 +``` + +If the target environment uses a metrics port other than `9253`, keep the existing port and remove only the IP binding. + +Restart the DaemonSet to apply the configuration: + +```bash +kubectl -n "$NS" rollout restart ds "$DS" +kubectl -n "$NS" rollout status ds "$DS" +``` + +Confirm that the Corefile is updated, and verify that metrics can be accessed from the monitoring collection path: + +```bash +kubectl -n "$NS" get cm "$CM" -o yaml | grep 'prometheus' +``` + ## Diagnostic Steps Confirm that NodeLocal DNSCache is installed and Pods are Ready: @@ -179,6 +229,12 @@ kubectl -n kube-system get cm node-local-dns -o yaml | grep 'health 127.0.0.1' kubectl -n kube-system get ds node-local-dns -o yaml | grep -A5 -E 'livenessProbe|readinessProbe' ``` +Confirm the metrics listen address in the Corefile: + +```bash +kubectl -n kube-system get cm node-local-dns -o yaml | grep 'prometheus' +``` + Confirm the DNS servers used by newly created Pods: ```bash @@ -204,6 +260,14 @@ sudo systemctl restart kubelet Then recreate the affected Pods so their `/etc/resolv.conf` is regenerated. +If changing the metrics listen address causes problems, restore the backed-up ConfigMap and restart the DaemonSet: + +```bash +kubectl apply -f node-local-dns-cm.backup.yaml +kubectl -n kube-system rollout restart ds/node-local-dns +kubectl -n kube-system rollout status ds/node-local-dns +``` + ## Related Information If the cluster is upgraded by rebuilding nodes, directly changing `/var/lib/kubelet/kubeadm-flags.env` on nodes is lost after node rebuild. You need to synchronize the same multi-address `cluster-dns` value to every `kubeletExtraArgs` location in the cluster template: @@ -212,4 +276,4 @@ If the cluster is upgraded by rebuilding nodes, directly changing `/var/lib/kube - `KubeadmControlPlane` → `joinConfiguration` → `nodeRegistration` → `kubeletExtraArgs` - `KubeadmConfigTemplate` → `template` → `spec` → `joinConfiguration` → `nodeRegistration` → `kubeletExtraArgs` -The long-term fix should be implemented on the product side, for example by exposing the health check port in the NodeLocal DNSCache plugin parameters, or by supporting multiple kubelet `cluster-dns` addresses in the plugin or cluster configuration. +The long-term fix should be implemented on the product side, for example by exposing the health check port and metrics listen address in the NodeLocal DNSCache plugin parameters, or by supporting multiple kubelet `cluster-dns` addresses in the plugin or cluster configuration. diff --git a/docs/zh/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md b/docs/zh/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md index 5510da5ed..9f516d170 100644 --- a/docs/zh/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md +++ b/docs/zh/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md @@ -7,7 +7,7 @@ ProductsVersion: - '4.2.x,4.3.x,4.4.x' --- -# NodeLocal DNSCache 8080 端口冲突和 DNS 单点风险的临时规避方案 +# NodeLocal DNSCache 端口冲突、DNS 单点和指标访问问题的临时规避方案 ## 问题 @@ -15,6 +15,7 @@ ProductsVersion: - `node-cache` Pod 使用 `hostNetwork: true`，并在节点 loopback `127.0.0.1:8080` 上暴露健康检查端点。如果节点上的业务进程、运维代理或 `hostNetwork` Pod 也绑定 `127.0.0.1:8080` 或 `0.0.0.0:8080`，会发生端口冲突。 - NodeLocal DNSCache 生效后，新建 Pod 使用节点本地 DNS 地址作为 DNS 服务器。当某个节点上的 `node-cache` Pod 不可用时，该节点上 Pod 的 DNS 解析可能失败，且不会自动无感切换到 CoreDNS。 +- 如果 Corefile 中的 `prometheus` 指令绑定到 NodeLocal DNSCache IP，例如 `169.254.20.10:9253`，外部监控或面板可能无法访问该 metrics 端点。 ## 根本原因 @@ -34,6 +35,8 @@ livenessProbe: 插件安装任务还会把 kubelet `--cluster-dns` 配置为 NodeLocal DNSCache IP。如果需要释放 `8080`，或把 CoreDNS ClusterIP 作为辅助 DNS server，需要临时修改运行中资源或节点 kubelet 配置。 +对于 metrics 访问问题，根因通常是 Corefile 中的 `prometheus` 指令绑定到了 NodeLocal DNSCache IP，而不是只监听端口。外部监控采集路径无法访问该节点本地地址时，就会拿不到指标。 + ## 解决方案 以下方案仅用于临时规避。插件升级、重装、平台调谐、chart 重新渲染或节点重建后，手工修改可能被覆盖。建议在变更窗口执行，并在修改前保留备份。 @@ -163,6 +166,53 @@ nameserver 10.96.0.10 如果集群启用了 NetworkPolicy，需要同时放行 Pod 访问 NodeLocal DNSCache IP 和 CoreDNS ClusterIP 的 TCP/UDP `53` 端口。 +### 方案三：去掉 Prometheus metrics 的固定 IP 绑定 + +该方案适用于外部监控或面板无法访问 NodeLocal DNSCache metrics 的场景。只修改 Corefile 中的 `prometheus` 监听地址，不修改 DNS 服务端口。 + +先备份当前 ConfigMap： + +```bash +NS=kube-system +CM=node-local-dns +DS=node-local-dns + +kubectl -n "$NS" get cm "$CM" -o yaml > node-local-dns-cm.backup.yaml +``` + +编辑 ConfigMap： + +```bash +kubectl -n "$NS" edit cm "$CM" +``` + +将 Corefile 中绑定固定 IP 的 `prometheus` 指令： + +```text +prometheus 169.254.20.10:9253 +``` + +修改为只监听端口： + +```text +prometheus :9253 +``` + +如果目标环境中的 metrics 端口不是 `9253`，保持现场端口不变，只去掉 IP 绑定。 + +重启 DaemonSet 使配置生效： + +```bash +kubectl -n "$NS" rollout restart ds "$DS" +kubectl -n "$NS" rollout status ds "$DS" +``` + +确认 Corefile 已更新，并从监控采集路径验证 metrics 可访问： + +```bash +kubectl -n "$NS" get cm "$CM" -o yaml | grep 'prometheus' +``` + ## 诊断步骤 确认 NodeLocal DNSCache 是否安装、Pod 是否就绪： @@ -179,6 +229,12 @@ kubectl -n kube-system get cm node-local-dns -o yaml | grep 'health 127.0.0.1' kubectl -n kube-system get ds node-local-dns -o yaml | grep -A5 -E 'livenessProbe|readinessProbe' ``` +确认 Corefile 中的 metrics 监听地址： + +```bash +kubectl -n kube-system get cm node-local-dns -o yaml | grep 'prometheus' +``` + 确认新建 Pod 实际使用的 DNS server： ```bash @@ -204,6 +260,14 @@ sudo systemctl restart kubelet 然后重建受影响 Pod，使其 `/etc/resolv.conf` 重新生成。 +如果修改 metrics 监听地址后异常，恢复备份的 ConfigMap，并重启 DaemonSet： + +```bash +kubectl apply -f node-local-dns-cm.backup.yaml +kubectl -n kube-system rollout restart ds/node-local-dns +kubectl -n kube-system rollout status ds/node-local-dns +``` + ## 相关说明 如果集群通过重建节点方式升级，直接修改节点上的 `/var/lib/kubelet/kubeadm-flags.env` 会在节点重建后丢失。需要把同样的多地址 `cluster-dns` 值同步到集群模板中的每一处 `kubeletExtraArgs`： @@ -212,4 +276,4 @@ sudo systemctl restart kubelet - `KubeadmControlPlane` → `joinConfiguration` → `nodeRegistration` → `kubeletExtraArgs` - `KubeadmConfigTemplate` → `template` → `spec` → `joinConfiguration` → `nodeRegistration` → `kubeletExtraArgs` -长期方案应在产品侧处理，例如在 NodeLocal DNSCache 插件参数中暴露健康检查端口，或在插件/集群配置中支持多个 kubelet `cluster-dns` 地址。 +长期方案应在产品侧处理，例如在 NodeLocal DNSCache 插件参数中暴露健康检查端口、metrics 监听地址，或在插件/集群配置中支持多个 kubelet `cluster-dns` 地址。 From fb4dd78e863457487c87d225368eb1e65523e966 Mon Sep 17 00:00:00 2001 From: clyi Date: Fri, 14 Aug 2026 14:12:36 +0800 Subject: [PATCH 4/4] docs(acp): restructure nodelocaldns workarounds by issue --- .../acp/NodeLocal_DNSCache_S2_Workarounds.md | 114 +++++++----------- .../acp/NodeLocal_DNSCache_S2_Workarounds.md | 114 +++++++----------- 2 files changed, 84 insertions(+), 144 deletions(-) diff --git a/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md b/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md index f7ec5dcfd..2157cbb4b 100644 --- a/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md +++ b/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md @@ -7,19 +7,21 @@ ProductsVersion: - '4.2.x,4.3.x,4.4.x' --- -# Temporary Workarounds for NodeLocal DNSCache Port Conflicts, DNS Single-Point Risk, and Metrics Access Issues +# Temporary Workarounds for Common NodeLocal DNSCache Issues in Field Environments -## Problem +This article provides temporary workarounds for three common NodeLocal DNSCache issues in field environments: -After NodeLocal DNSCache is enabled in an ACP cluster, the following risks may occur: +- Health check port `8080` conflicts. +- DNS resolution is affected when the `node-cache` Pod on a node is unavailable. +- External monitoring systems or dashboards cannot collect Prometheus metrics because the metrics endpoint binds to the NodeLocal DNSCache IP. -- The `node-cache` Pod runs with `hostNetwork: true` and exposes its health check endpoint on the node loopback `127.0.0.1:8080`. If a business process, operations agent, or `hostNetwork` Pod on the same node also binds `127.0.0.1:8080` or `0.0.0.0:8080`, a port conflict occurs. -- After NodeLocal DNSCache takes effect, newly created Pods use the node-local DNS address as their DNS server. If the `node-cache` Pod on a node is unavailable, DNS resolution for Pods on that node may fail and does not automatically switch to CoreDNS without impact. -- If the Corefile `prometheus` directive binds to the NodeLocal DNSCache IP, for example `169.254.20.10:9253`, external monitoring systems or dashboards may fail to access the metrics endpoint. +These workarounds are temporary. Manual changes may be overwritten after plugin upgrade, plugin reinstall, platform reconciliation, chart re-rendering, or node rebuild. Perform the change in a maintenance window and keep backups before editing resources. -## Root Cause +## Issue 1: NodeLocal DNSCache health check uses port 8080 -The current NodeLocal DNSCache plugin exposes only the NodeLocal DNS IP parameter. It does not expose the health check port or multiple DNS server settings for kubelet. The generated Corefile and DaemonSet probe use `8080` by default: +**Symptom:** After NodeLocal DNSCache is enabled, a business process, operations agent, or `hostNetwork` Pod on the node cannot bind `127.0.0.1:8080` or `0.0.0.0:8080`. + +**Cause:** The `node-cache` Pod runs with `hostNetwork: true` and exposes its health check endpoint on the node loopback `127.0.0.1:8080`. The current plugin does not expose the health check port. The generated Corefile and DaemonSet probe use `8080` by default. ```text health 127.0.0.1:8080 @@ -33,17 +35,7 @@ livenessProbe: port: 8080 ``` -The plugin installation job also configures kubelet `--cluster-dns` to the NodeLocal DNSCache IP. To release port `8080`, or to use CoreDNS ClusterIP as an additional DNS server, you need to temporarily modify runtime resources or node kubelet configuration. - -For metrics access issues, the common root cause is that the Corefile `prometheus` directive binds to the NodeLocal DNSCache IP instead of listening only on a port. If the external monitoring path cannot reach that node-local address, metrics cannot be collected. - -## Resolution - -The following workarounds are temporary. Manual changes may be overwritten after plugin upgrade, plugin reinstall, platform reconciliation, chart re-rendering, or node rebuild. Perform the change in a maintenance window and keep backups before editing resources. - -### Workaround 1: Change the NodeLocal DNSCache health check port - -Use this workaround when node-local port `8080` must be released. Change both the Corefile `health` port and the DaemonSet probe port. The two values must stay consistent. +**Resolution:** Change both the Corefile `health` port and the DaemonSet probe port. The two values must stay consistent. Confirm resource names and back up current resources: @@ -87,7 +79,7 @@ livenessProbe: port: 18080 ``` -If the target environment also has a `readinessProbe` that accesses `/health` or `8080`, change it as well. Do not change DNS service port `53` or metrics port `9353`. +If the target environment also has a `readinessProbe` that accesses `/health` or `8080`, change it as well. Do not change DNS service port `53` or the metrics port used in the environment. This issue only requires changing the health check port. Wait for the DaemonSet rolling update to complete and verify DNS resolution: @@ -105,9 +97,21 @@ ss -ltnp | grep ':8080' The expected result is that `18080` is listened on by NodeLocal DNSCache, and `8080` is no longer listened on by NodeLocal DNSCache. -### Workaround 2: Configure CoreDNS ClusterIP as an additional DNS server +**Rollback:** If changing the health check port causes problems, restore the backed-up ConfigMap and DaemonSet: + +```bash +kubectl apply -f node-local-dns-cm.backup.yaml +kubectl apply -f node-local-dns-ds.backup.yaml +kubectl -n kube-system rollout status ds/node-local-dns +``` + +## Issue 2: DNS resolution fails when the `node-cache` Pod is unavailable + +**Symptom:** After NodeLocal DNSCache takes effect, newly created Pods use the node-local DNS address as their DNS server. If the `node-cache` Pod on a node becomes unavailable, is evicted, or restarts during an upgrade, DNS resolution for Pods on that node may fail. + +**Cause:** The plugin installation job configures kubelet `--cluster-dns` to the NodeLocal DNSCache IP. By default, newly created Pods have only the NodeLocal DNSCache IP in `/etc/resolv.conf`, without the CoreDNS ClusterIP as an additional DNS server. -Use this workaround when you want to reduce the single-point impact of NodeLocal DNSCache. After configuration, newly created Pods have both the NodeLocal DNSCache IP and CoreDNS ClusterIP in `/etc/resolv.conf`. +**Resolution:** Configure CoreDNS ClusterIP as an additional DNS server for kubelet. After configuration, newly created Pods have both the NodeLocal DNSCache IP and CoreDNS ClusterIP in `/etc/resolv.conf`. This is not a transparent failover mechanism. DNS resolver retry behavior differs between business images. When the first DNS server is unavailable, some workloads may wait for timeout before trying the next DNS server, which can slow down DNS resolution during the failure. @@ -166,9 +170,22 @@ nameserver 10.96.0.10 If NetworkPolicy is enabled in the cluster, allow Pods to access both the NodeLocal DNSCache IP and CoreDNS ClusterIP on TCP/UDP port `53`. -### Workaround 3: Remove the fixed IP binding from Prometheus metrics +**Rollback:** If configuring multiple DNS servers causes problems, log in to the modified nodes, restore `/var/lib/kubelet/kubeadm-flags.env` from the backup, and restart kubelet: -Use this workaround when external monitoring systems or dashboards cannot access NodeLocal DNSCache metrics. Change only the `prometheus` listen address in the Corefile. Do not change the DNS service port. +```bash +sudo cp -a /var/lib/kubelet/kubeadm-flags.env.bak. /var/lib/kubelet/kubeadm-flags.env +sudo systemctl restart kubelet +``` + +Then recreate the affected Pods so their `/etc/resolv.conf` is regenerated. + +## Issue 3: Prometheus metrics bind to a fixed NodeLocal DNSCache IP and cannot be collected externally + +**Symptom:** External monitoring systems or dashboards cannot access NodeLocal DNSCache metrics. + +**Cause:** The Corefile `prometheus` directive may bind to the NodeLocal DNSCache IP, for example `169.254.20.10:9253`. If the external monitoring collection path cannot reach that node-local address, metrics cannot be collected. + +**Resolution:** Change only the `prometheus` listen address in the Corefile. Do not change the DNS service port. Back up the current ConfigMap first: @@ -213,54 +230,7 @@ Confirm that the Corefile is updated, and verify that metrics can be accessed fr kubectl -n "$NS" get cm "$CM" -o yaml | grep 'prometheus' ``` -## Diagnostic Steps - -Confirm that NodeLocal DNSCache is installed and Pods are Ready: - -```bash -kubectl -n kube-system get pods -l k8s-app=node-local-dns -o wide -kubectl -n kube-system rollout status ds/node-local-dns -``` - -Confirm whether the current Corefile and DaemonSet probe still use `8080`: - -```bash -kubectl -n kube-system get cm node-local-dns -o yaml | grep 'health 127.0.0.1' -kubectl -n kube-system get ds node-local-dns -o yaml | grep -A5 -E 'livenessProbe|readinessProbe' -``` - -Confirm the metrics listen address in the Corefile: - -```bash -kubectl -n kube-system get cm node-local-dns -o yaml | grep 'prometheus' -``` - -Confirm the DNS servers used by newly created Pods: - -```bash -kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- cat /etc/resolv.conf -``` - -## Rollback - -If changing the health check port causes problems, restore the backed-up ConfigMap and DaemonSet: - -```bash -kubectl apply -f node-local-dns-cm.backup.yaml -kubectl apply -f node-local-dns-ds.backup.yaml -kubectl -n kube-system rollout status ds/node-local-dns -``` - -If configuring multiple DNS servers causes problems, log in to the modified nodes, restore `/var/lib/kubelet/kubeadm-flags.env` from the backup, and restart kubelet: - -```bash -sudo cp -a /var/lib/kubelet/kubeadm-flags.env.bak. /var/lib/kubelet/kubeadm-flags.env -sudo systemctl restart kubelet -``` - -Then recreate the affected Pods so their `/etc/resolv.conf` is regenerated. - -If changing the metrics listen address causes problems, restore the backed-up ConfigMap and restart the DaemonSet: +**Rollback:** If changing the metrics listen address causes problems, restore the backed-up ConfigMap and restart the DaemonSet: ```bash kubectl apply -f node-local-dns-cm.backup.yaml diff --git a/docs/zh/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md b/docs/zh/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md index 9f516d170..e74af35fb 100644 --- a/docs/zh/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md +++ b/docs/zh/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md @@ -7,19 +7,21 @@ ProductsVersion: - '4.2.x,4.3.x,4.4.x' --- -# NodeLocal DNSCache 端口冲突、DNS 单点和指标访问问题的临时规避方案 +# NodeLocal DNSCache 常见现场问题的临时规避方案 -## 问题 +本文提供 NodeLocal DNSCache 三类常见现场问题的临时规避方案： -在 ACP 集群中启用 NodeLocal DNSCache 后，可能出现以下风险： +- 健康检查端口 `8080` 冲突。 +- `node-cache` Pod 异常后，节点上新建或存量 Pod 的 DNS 解析受影响。 +- Prometheus metrics 绑定到 NodeLocal DNSCache IP 后，外部监控或面板无法采集。 -- `node-cache` Pod 使用 `hostNetwork: true`，并在节点 loopback `127.0.0.1:8080` 上暴露健康检查端点。如果节点上的业务进程、运维代理或 `hostNetwork` Pod 也绑定 `127.0.0.1:8080` 或 `0.0.0.0:8080`，会发生端口冲突。 -- NodeLocal DNSCache 生效后，新建 Pod 使用节点本地 DNS 地址作为 DNS 服务器。当某个节点上的 `node-cache` Pod 不可用时，该节点上 Pod 的 DNS 解析可能失败，且不会自动无感切换到 CoreDNS。 -- 如果 Corefile 中的 `prometheus` 指令绑定到 NodeLocal DNSCache IP，例如 `169.254.20.10:9253`，外部监控或面板可能无法访问该 metrics 端点。 +这些方案仅用于临时规避。插件升级、重装、平台调谐、chart 重新渲染或节点重建后，手工修改可能被覆盖。建议在变更窗口执行，并在修改前保留备份。 -## 根本原因 +## 问题 1：NodeLocal DNSCache 健康检查端口占用 8080 -当前 NodeLocal DNSCache 插件只暴露 NodeLocal DNS IP 参数，未暴露健康检查端口或 kubelet 多 DNS server 参数。插件生成的 Corefile 和 DaemonSet 探针默认使用 `8080`： +**现象：** 启用 NodeLocal DNSCache 后，节点上的业务进程、运维代理或 `hostNetwork` Pod 无法绑定 `127.0.0.1:8080` 或 `0.0.0.0:8080`。 + +**原因：** `node-cache` Pod 使用 `hostNetwork: true`，并在节点 loopback `127.0.0.1:8080` 上暴露健康检查端点。当前插件未暴露健康检查端口参数，生成的 Corefile 和 DaemonSet 探针默认使用 `8080`。 ```text health 127.0.0.1:8080 @@ -33,17 +35,7 @@ livenessProbe: port: 8080 ``` -插件安装任务还会把 kubelet `--cluster-dns` 配置为 NodeLocal DNSCache IP。如果需要释放 `8080`，或把 CoreDNS ClusterIP 作为辅助 DNS server，需要临时修改运行中资源或节点 kubelet 配置。 - -对于 metrics 访问问题，根因通常是 Corefile 中的 `prometheus` 指令绑定到了 NodeLocal DNSCache IP，而不是只监听端口。外部监控采集路径无法访问该节点本地地址时，就会拿不到指标。 - -## 解决方案 - -以下方案仅用于临时规避。插件升级、重装、平台调谐、chart 重新渲染或节点重建后，手工修改可能被覆盖。建议在变更窗口执行，并在修改前保留备份。 - -### 方案一：修改 NodeLocal DNSCache 健康检查端口 - -该方案适用于必须释放节点本地 `8080` 端口的场景。需要同时修改 Corefile 中的 `health` 端口和 DaemonSet 探针端口，两个位置必须保持一致。 +**解决方案：** 同时修改 Corefile 中的 `health` 端口和 DaemonSet 探针端口，两个位置必须保持一致。 先确认资源名称，并备份当前资源： @@ -87,7 +79,7 @@ livenessProbe: port: 18080 ``` -如果目标环境中还存在访问 `/health` 或 `8080` 的 `readinessProbe`，也需要同步修改。不要修改 DNS 服务端口 `53` 或 metrics 端口 `9353`。 +如果目标环境中还存在访问 `/health` 或 `8080` 的 `readinessProbe`，也需要同步修改。不要修改 DNS 服务端口 `53` 或现场 metrics 端口；本问题只处理健康检查端口。 等待 DaemonSet 滚动更新完成，并验证 DNS 解析正常： @@ -105,9 +97,21 @@ ss -ltnp | grep ':8080' 预期 `18080` 由 NodeLocal DNSCache 监听，`8080` 不再由 NodeLocal DNSCache 监听。 -### 方案二：配置 CoreDNS ClusterIP 作为辅助 DNS server +**回滚：** 如果修改健康检查端口后异常，恢复备份的 ConfigMap 和 DaemonSet： + +```bash +kubectl apply -f node-local-dns-cm.backup.yaml +kubectl apply -f node-local-dns-ds.backup.yaml +kubectl -n kube-system rollout status ds/node-local-dns +``` + +## 问题 2：`node-cache` Pod 异常后节点 DNS 解析失败 + +**现象：** NodeLocal DNSCache 生效后，新建 Pod 使用节点本地 DNS 地址作为 DNS 服务器。当某个节点上的 `node-cache` Pod 异常、被驱逐或升级重启时，该节点上 Pod 的 DNS 解析可能失败。 + +**原因：** 插件安装任务会把 kubelet `--cluster-dns` 配置为 NodeLocal DNSCache IP。默认情况下，新建 Pod 的 `/etc/resolv.conf` 只有 NodeLocal DNSCache IP，没有 CoreDNS ClusterIP 作为辅助 DNS server。 -该方案适用于希望降低 NodeLocal DNSCache 单点影响的场景。配置后，新建 Pod 的 `/etc/resolv.conf` 中同时包含 NodeLocal DNSCache IP 和 CoreDNS ClusterIP。 +**解决方案：** 将 CoreDNS ClusterIP 配置为 kubelet 的辅助 DNS server。配置后，新建 Pod 的 `/etc/resolv.conf` 中同时包含 NodeLocal DNSCache IP 和 CoreDNS ClusterIP。 该方案不是无感故障切换机制。不同业务镜像中的 DNS 解析器重试行为不同；当第一个 DNS server 不可用时，部分工作负载可能需要等待超时后才尝试下一个 DNS server，故障期间 DNS 解析可能变慢。 @@ -166,9 +170,22 @@ nameserver 10.96.0.10 如果集群启用了 NetworkPolicy，需要同时放行 Pod 访问 NodeLocal DNSCache IP 和 CoreDNS ClusterIP 的 TCP/UDP `53` 端口。 -### 方案三：去掉 Prometheus metrics 的固定 IP 绑定 +**回滚：** 如果配置多个 DNS server 后异常，登录已修改节点，将 `/var/lib/kubelet/kubeadm-flags.env` 恢复为备份文件并重启 kubelet： -该方案适用于外部监控或面板无法访问 NodeLocal DNSCache metrics 的场景。只修改 Corefile 中的 `prometheus` 监听地址，不修改 DNS 服务端口。 +```bash +sudo cp -a /var/lib/kubelet/kubeadm-flags.env.bak. /var/lib/kubelet/kubeadm-flags.env +sudo systemctl restart kubelet +``` + +然后重建受影响 Pod，使其 `/etc/resolv.conf` 重新生成。 + +## 问题 3：Prometheus metrics 绑定固定 NodeLocal DNSCache IP 后无法被外部采集 + +**现象：** 外部监控或面板无法访问 NodeLocal DNSCache metrics。 + +**原因：** Corefile 中的 `prometheus` 指令可能绑定到了 NodeLocal DNSCache IP，例如 `169.254.20.10:9253`。如果外部监控采集路径无法访问该节点本地地址，就会拿不到指标。 + +**解决方案：** 只修改 Corefile 中的 `prometheus` 监听地址，不修改 DNS 服务端口。 先备份当前 ConfigMap： @@ -213,54 +230,7 @@ kubectl -n "$NS" rollout status ds "$DS" kubectl -n "$NS" get cm "$CM" -o yaml | grep 'prometheus' ``` -## 诊断步骤 - -确认 NodeLocal DNSCache 是否安装、Pod 是否就绪： - -```bash -kubectl -n kube-system get pods -l k8s-app=node-local-dns -o wide -kubectl -n kube-system rollout status ds/node-local-dns -``` - -确认当前 Corefile 和 DaemonSet 探针是否仍使用 `8080`： - -```bash -kubectl -n kube-system get cm node-local-dns -o yaml | grep 'health 127.0.0.1' -kubectl -n kube-system get ds node-local-dns -o yaml | grep -A5 -E 'livenessProbe|readinessProbe' -``` - -确认 Corefile 中的 metrics 监听地址： - -```bash -kubectl -n kube-system get cm node-local-dns -o yaml | grep 'prometheus' -``` - -确认新建 Pod 实际使用的 DNS server： - -```bash -kubectl run dns-check --rm -it --restart=Never --image=busybox:1.36 -- cat /etc/resolv.conf -``` - -## 回滚 - -如果修改健康检查端口后异常，恢复备份的 ConfigMap 和 DaemonSet： - -```bash -kubectl apply -f node-local-dns-cm.backup.yaml -kubectl apply -f node-local-dns-ds.backup.yaml -kubectl -n kube-system rollout status ds/node-local-dns -``` - -如果配置多个 DNS server 后异常，登录已修改节点，将 `/var/lib/kubelet/kubeadm-flags.env` 恢复为备份文件并重启 kubelet： - -```bash -sudo cp -a /var/lib/kubelet/kubeadm-flags.env.bak. /var/lib/kubelet/kubeadm-flags.env -sudo systemctl restart kubelet -``` - -然后重建受影响 Pod，使其 `/etc/resolv.conf` 重新生成。 - -如果修改 metrics 监听地址后异常，恢复备份的 ConfigMap，并重启 DaemonSet： +**回滚：** 如果修改 metrics 监听地址后异常，恢复备份的 ConfigMap，并重启 DaemonSet： ```bash kubectl apply -f node-local-dns-cm.backup.yaml