Talos
OmniNetworking
KubeSpan
Learn to use KubeSpan to connect Talos Linux machines securely across networks.
KubeSpan is a feature of Talos that automates the setup and maintenance of a full mesh WireGuard network for your cluster, giving you the ability to operate hybrid Kubernetes clusters that can span the edge, datacenter, and cloud.
Management of keys and discovery of peers can be completely automated, making it simple and easy to create hybrid clusters.
KubeSpan consists of client code in Talos Linux, as well as a discovery service that enables clients to securely find each other.
Sidero Labs operates a free Discovery Service, but the discovery service may, with a commercial license, be operated by your organization and can be downloaded here.
The setting
Talos automatically configures unique IPv6 address for each node in the cluster-specific IPv6 ULA prefix.
The Wireguard private key is generated and never leaves the node, while the public key is published through the cluster discovery.
The peer ID is the Wireguard public key.
KubeSpan peer status includes following information:
The endpoint ID is the base64 encoded WireGuard public key.
The observed endpoints are submitted back to the discovery service (if enabled) so that other peers can try additional endpoints to establish the connection.
Video Walkthrough
To see a live demo of KubeSpan, see one the videos below:Network Requirements
KubeSpan uses UDP port 51820 to carry all KubeSpan encrypted traffic. Because UDP traversal of firewalls is often lenient, and the Discovery Service communicates the apparent IP address of all peers to all other peers, KubeSpan will often work automatically, even when each nodes is behind their own firewall. However, when both ends of a KubeSpan connection are behind firewalls, it is possible the connection may not be established correctly - it depends on each end sending out packets in a limited time window. Thus best practice is to ensure that one end of all possible node-node communication allows UDP port 51820, inbound. For example, if control plane nodes are running in a corporate data center, behind firewalls, KubeSpan connectivity will work correctly so long as worker nodes on the public Internet can receive packets on UDP port 51820. (Note the workers will also need to receive TCP port 50000 for initial configuration viatalosctl).
An alternative topology would be to run control plane nodes in a public cloud, and allow inbound UDP port 51820 to the control plane nodes.
Workers could be behind firewalls, and KubeSpan connectivity will be established.
Note that if workers are in different locations, behind different firewalls, the KubeSpan connectivity between workers should be correctly established, but may require opening the KubeSpan UDP port on the local firewall also.
Caveats
Kubernetes API Endpoint Limitations
When the K8s endpoint is an IP address that is not part of Kubespan, but is an address that is forwarded on to the Kubespan address of a control plane node, without changing the source address, then worker nodes will fail to join the cluster. In such a case, the control plane node has no way to determine whether the packet arrived on the private Kubespan address, or the public IP address. If the source of the packet was a Kubespan member, the reply will be Kubespan encapsulated, and thus not translated to the public IP, and so the control plane will reply to the session with the wrong address. This situation is seen, for example, when the Kubernetes API endpoint is the public IP of a VM in GCP or Azure for a single node control plane. The control plane will receive packets on the public IP, but will reply from it’s KubeSpan address. The workaround is to create a load balancer to terminate the Kubernetes API endpoint.Digital Ocean Limitations
Digital Ocean assigns an “Anchor IP” address to each droplet. Talos Linux correctly identifies this as a link-local address, and configures KubeSpan correctly, but this address will often be selected by Flannel or other CNIs as a node’s private IP. Because this address is not routable, nor advertised via KubeSpan, it will break pod-pod communication between nodes. This can be worked-around by assigning a non-Anchor private IP:kubectl annotate node do-worker flannel.alpha.coreos.com/public-ip-overwrite=10.116.X.X
Then restarting flannel:
kubectl delete pods -n kube-system -l k8s-app=flannel
Host Port Limitations
As mentioned in Network Requirements, Kubespan uses UDP port 51820 to carry all KubeSpan encrypted traffic. For clusters that make heavy use of host ports for Kubernetes pods, care should be taken to ensure that this port is not given to these pods. Failure to do so can result in a pod being assigned the 51820 port and conflicting with Kubespan traffic.Enabling
Creating a New Cluster
To enable KubeSpan for a new cluster, we can use the--with-kubespan flag in talosctl gen config.
This will enable peer discovery and KubeSpan.
The default discovery service is an external service hosted by Sidero Labs at https://discovery.talos.dev/.
Contact Sidero Labs if you need to run this service privately.
Enabling for an Existing Cluster
In order to enable KubeSpan on an existing cluster, enablekubespan and discovery settings in the machine config for each machine in the cluster (discovery is enabled by default):
Configuration
KubeSpan will automatically discover all cluster members, exchange Wireguard public keys and establish a full mesh network. There are configuration options available which are not usually required:advertiseKubernetesNetworks controls whether the node will advertise Kubernetes service and pod networks to other nodes in the cluster over KubeSpan.
It defaults to being disabled, which means KubeSpan only controls the node-to-node traffic, while pod-to-pod traffic is routed and encapsulated by CNI.
This setting should not be enabled with Calico and Cilium CNI plugins, as they do their own pod IP allocation which is not visible to KubeSpan.
The setting allowDownPeerBypass controls whether the node will allow traffic to bypass WireGuard if the destination is not connected over KubeSpan.
If enabled, there is a risk that traffic will be routed unencrypted if the destination is not connected over KubeSpan, but it allows a workaround
for the case where a node is not connected to the KubeSpan network, but still needs to access the cluster.
The mtu setting configures the Wireguard MTU, which defaults to 1420.
This default value of 1420 is safe to use when the underlying network MTU is 1500, but if the underlying network MTU is smaller, the KubeSpanMTU should be adjusted accordingly:
KubeSpanMTU = UnderlyingMTU - 80.
The filters setting allows hiding some endpoints from being advertised over KubeSpan.
This is useful when some endpoints are known to be unreachable between the nodes, so that KubeSpan doesn’t try to establish a connection to them.
Another use-case is hiding some endpoints if nodes can connect on multiple networks, and some of the networks are more preferable than others.
To include additional announced endpoints, such as inbound NAT mappings, you can add the machine config document.
Resource Definitions
KubeSpanIdentities
A node’s WireGuard identities can be obtained with:KubeSpanIdentity is persisted across reboots and upgrades in STATE partition in the file kubespan-identity.yaml.
KubeSpanPeerSpecs
A node’s WireGuard peers can be obtained with:KubeSpanPeerSpecs are built from the cluster discovery data.
KubeSpanPeerStatuses
The status of a node’s WireGuard peers can be obtained with:- the actual endpoint used for peer communication
- link state:
unknown: the endpoint was just changed, link state is not known yetup: there is a recent handshake from the peerdown: there is no handshake from the peer
- number of bytes sent/received over the Wireguard link with the peer
down, Talos will be cycling through the available endpoints until it finds the one which works.
Peer status information is updated every 30 seconds.