DNS options in the Kind Config

[Kukkerem]

What would you like to be added:
A new section in the Kind Config that allows for setting Docker network options, specifically the --dns option. We want to specify the DNS server as one of the already running Kind clusters.

Proposed configuration:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  dns:
    nameservers:
    - "172.18.0.100"  # Custom DNS server IP

Why is this needed:
We are building a multi-cluster development environment around Kind, with a local DNS in one of the clusters. This DNS is responsible for routing between clusters. Currently, the Kubernetes API server uses the host's DNS configuration, which creates issues when:

• Admission webhooks must resolve services in other Kind clusters by DNS name.
• Host DNS modification is not desirable or possible in CI/CD environments.

Modifying the local /etc/resolv.conf inside running containers is fragile, and we don't want to use IP addresses to avoid breaking the abstraction layer.

[aojea]

We want to specify the DNS server as one of the already running Kind clusters.

we use the "magic" DNS from docker/podman to get reboot/restart to work since they provide container/node name resolution.

Admission webhooks must resolve services in other Kind clusters by DNS name.

this seems to be a very tightly coupled request to a very specific environment solutions, so you want to resolve multi cluster Services, and in order to do that you want to point the container to use that specific cluster dns server.

We build kind to mimic the real cluster behaviors, so trying to shortcut those by leveraging the docker properties does not seem the correct answer. How this problem will be solved in a real environment? external dns maybe?

[Kukkerem]

You're right that external DNS would be the production solution. However, we can't easily set up external DNS infrastructure for our testing. We test a multi-cluster solution locally and in CI/CD.

In this sense, we also want to replicate a real-life behaviour, where you can configure custom DNS on the node level. This modification would allow Kind to replicate the node-level DNS configuration in actual deployments.

[aojea]

This modification would allow Kind to replicate the node-level DNS configuration in actual deployments.

For replicate actual deployments you configure your host resolv.conf, using ansible, puppet, chef, ...

Independently, as commented previously, the nodes restart depend on the dns knowing the node names, and is an example on why coupling kind to the docker details is not ideal

kind/images/base/files/usr/local/bin/entrypoint

Lines 446 to 470 in f36d008

	enable_network_magic(){
	# well-known docker embedded DNS is at 127.0.0.11:53
	local docker_embedded_dns_ip='127.0.0.11'
	# first we need to detect an IP to use for reaching the docker host
	local docker_host_ip
	docker_host_ip="$( (head -n1 <(timeout 5 getent ahostsv4 'host.docker.internal') | cut -d' ' -f1) || true)"
	# if the ip doesn't exist or is a loopback address use the default gateway
	if [[ -z "${docker_host_ip}" ]] || [[ $docker_host_ip =~ ^127\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
	docker_host_ip=$(ip -4 route show default | cut -d' ' -f3)
	fi
	# patch docker's iptables rules to switch out the DNS IP
	iptables-save \
	| sed \
	`# switch docker DNS DNAT rules to our chosen IP` \
	-e "s/-d ${docker_embedded_dns_ip}/-d ${docker_host_ip}/g" \
	`# we need to also apply these rules to non-local traffic (from pods)` \
	-e 's/-A OUTPUT \(.*\) -j DOCKER_OUTPUT/\0\n-A PREROUTING \1 -j DOCKER_OUTPUT/' \
	`# switch docker DNS SNAT rules rules to our chosen IP` \
	-e "s/--to-source :53/--to-source ${docker_host_ip}:53/g"\
	`# nftables incompatibility between 1.8.8 and 1.8.7 omit the --dport flag on DNAT rules` \
	`# ensure --dport on DNS rules, due to https://github.com/kubernetes-sigs/kind/issues/3054` \
	-e "s/p -j DNAT --to-destination ${docker_embedded_dns_ip}/p --dport 53 -j DNAT --to-destination ${docker_embedded_dns_ip}/g" \
	| iptables-restore

[BenTheElder]

this DNS is responsible for routing between clusters.

FWIW, docker will already make it possible to resolve the nodes between the clusters, and as mentioned above we depend on resolving the nodes.

I think this feature is probably not a very good fit at the moment. If your application pods are not running with host network, then you can customize the coreDNS configmap within the cluster or the pod's DNS settings (e.g. with a mutating admission policy) in a slightly more portable way:

https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/