Kairos edgevpn with static IP and large subnet

[tri-bao]

Hello,
I'm new to Kairos. I'd love to hear your advice.

We are setting up a platform that extends native containerized application orchestration capabilities to hosts at Edge using KubeEdge. KubeEdge has 2 parts: CloudCore and EdgeCore. CloudCore will be installed on the Control Plane. EdgeCore will be installed on every Edge node. We try to use Karios for all nodes. Kubernetes control plane will be set up with K3s. Edge nodes will be set up without k3s. Kairos's automatic cluster co-ordination couldn't be used in this case because we don't run k3s on edge node (no kubelet on edge node).

The CloudCore will be installed with something like: keadm init --advertise-address="THE-EXPOSED-IP" ...

The EdgeCore will be installed with something like: keadm join --cloudcore-ipport="THE-EXPOSED-IP":10000

As we don't have support from Karios auto co-ordination, what we plan to do is:

• Have all nodes participate in one edgevpn
• Use static address on all control plane nodes:

p2p:
  vpn:
    env:
      # static IP for master node
      DHCP: "false"
      # first node 10.0.0.1/24, second node 10.0.0.2/24, third node: 10.0.0.3/24
      ADDRESS: "10.0.0.1/24"

• Use DHCP on all edge nodes

p2p:
  vpn:
    env:
      # dhcp for edge nodes
      DHCP: "true"
      # dhcp starts with 10.0.0.100
      ADDRESS: "10.0.0.100/24"

• Since master nodes IPs are known. We can hope we can join the edge node without worrying about the underlying network

I stood up one master node, test the VPN. I realized something that my not work for us: when edgevpn starts with the init address 10.0.0.100/24 it will create the following interface

51: kairos0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1200 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 10.0.0.100/24 brd 10.0.0.255 scope global kairos0
       valid_lft forever preferred_lft forever

and route:

10.0.0.0/24 dev kairos0 proto kernel scope link src 10.0.0.100

Looking at the route subnet /24. Nodes with assigned IP like 10.1.0.1 won't be connected (for sure, our nodes will exceed /24 space). I tested by running sudo ~/bin/kairosctl bridge --address=10.1.0.1/24 --token=.... and I am unable to ssh to the master node.

Did I mise use anything? Or do you see that there are other possibilities for our use case?

Thank you,
Bao

[jimmykarily]

Really nice idea @tri-bao! I'm afraid I don't have a clear answer to your questions but only 2 hints.

I think this is where the CIDR size is set: https://github.com/mudler/edgevpn/blob/45ace51385b5aad01ad8712853c90d559a339ee4/pkg/vpn/dhcp.go#L162-L170 . Maybe we can make that configurable if more IPs is the problem. But to test your setup, this shouldn't block you right? I mean, you should at least get some nodes connected before you run out of IP addresses.

Second one, As your nodes come up, they shouldn't assume that the other VPN nodes are immediately accessible. edgevpn takes some time to discover the other nodes. Maybe the KubeEdge services start too early, before the vpn connection has been established?

[tri-bao]

Thank you for your response, @jimmykarily After several tries, indeed, it works as expected for me. The /24 in the source code misleads me. I just need to start the static-IP node with address 10.0.0.1/8 and start dhcp-enabled nodes as 10.0.0.100/8. Routes and interface address on all nodes work as expected, e.g. /8 subnet.

BTW, I learned that, when modifying edgevpn config in file /oem/90_custom.yaml`, I have to reboot the machine twice in order to have it take effect. The first reboot is to generate the edgevpn-kairos.env file, then edgevpn service will actually use that file in the second reboot :)

About KubeEdge has to wait for edgeVPN to establish the connection, I'll have to test it. I believe it will be OK as the reason I want to use KubeEdge is that nodes can be offline/disconnected in an arbitrary duration (network disruption, power cut...)