# Interactive Dashboard
Source: https://docs.siderolabs.com/talos/v1.13/deploy-and-manage-workloads/interactive-dashboard
A tool to inspect the running Talos machine state on the physical video console.
Interactive dashboard is enabled for all Talos platforms except for SBC images.
The dashboard can be disabled with kernel parameter `talos.dashboard.disabled=1`.
The dashboard runs only on the physical video console (not serial console) on the 2nd virtual TTY.
The first virtual TTY shows kernel logs same as in Talos `<1.4.0>`.
The virtual TTYs can be switched with `` and `` keys.
Keys `` - `` can be used to switch between different screens of the dashboard.
The dashboard is using either UEFI framebuffer or VGA/VESA framebuffer (for legacy BIOS boot).
## Dashboard resolution control
On legacy BIOS systems, the screen resolution can be adjusted with the [`vga=` kernel parameter](https://docs.kernel.org/fb/vesafb.html).
In modern kernels and platforms, this parameter is often ignored. For reliable results, it is recommended to boot with **UEFI**.
When running in **UEFI mode**, you can set the screen resolution through your hypervisor or UEFI firmware settings.
## Summary screen (`F1`)
![Interactive Dashboard Summary Screen]()
The header shows brief information about the node:
* hostname
* Talos version
* uptime
* CPU and memory hardware information
* CPU and memory load, number of processes
Table view presents summary information about the machine:
* UUID (from SMBIOS data)
* Cluster name (when the machine config is available)
* Machine stage: `Installing`, `Upgrading`, `Booting`, `Maintenance`, `Running`, `Rebooting`, `Shutting down`, etc.
* Machine stage readiness: checks Talos service status, static pod status, etc. (for `Running` stage)
* Machine type: controlplane/worker
* Number of members discovered in the cluster
* Kubernetes version
* Status of Kubernetes components: `kubelet` and Kubernetes controlplane components (only on `controlplane` machines)
* Network information: Hostname, Addresses, Gateway, Connectivity, DNS and NTP servers
Bottom part of the screen shows kernel logs, same as on the virtual TTY 1.
## Monitor screen (`F2`)
![Interactive Dashboard Summary Screen]()
Monitor screen provides live view of the machine resource usage: CPU, memory, disk, network and processes.
## Network config screen (`F3`)
> Note: network config screen is only available for `metal` platform.
![Interactive Dashboard Summary Screen]()
Network config screen provides editing capabilities for the `metal` [platform network configuration](../platform-specific-installations/bare-metal-platforms/network-config).
The screen is split into three sections:
* the leftmost section provides a way to enter network configuration: hostname, DNS and NTP servers, configure the network interface either via DHCP or static IP address, etc.
* the middle section shows the current network configuration.
* the rightmost section shows the network configuration which will be applied after pressing "Save" button.
Once the platform network configuration is saved, it is immediately applied to the machine.
## Resource Viewer (`F4`/`F3`)
![Interactive Dashboard Resource Viewer Screen]()
Resource viewer screen provides a way to inspect the running Talos machine state when using the video console is the only option (e.g. when the network is not working and `talosctl` cannot be used).
When using the console, only non-sensitive resources are available, while when using `talosctl dashboard` command, available resources depend on the RBAC permissions of the user.
Resource viewer can be used to inspect network links, hardware devices, disks, etc.
The list of resources can be filtered by pressing `/` key and entering the search query. Use `/` to clear or change the filter.
Pressing `Enter` on a resource type shows a list of available resources of this type, and pressing `Enter` on a specific resource shows detailed information about it in YAML format.
# Scale down a Talos cluster
Source: https://docs.siderolabs.com/talos/v1.13/deploy-and-manage-workloads/scaling-down
How to remove nodes from a Talos Linux cluster.
To remove nodes from a Talos Linux cluster:
* `talosctl -n reset`
* `kubectl delete node `
The command `talosctl reset` will cordon and drain the node, leaving `etcd` if required, and then erase its disks and power down the system.
This command will also remove the node from registration with the discovery service, so it will no longer show up in `talosctl get members`.
It is still necessary to remove the node from Kubernetes, as noted above.
# Scale up a Talos cluster
Source: https://docs.siderolabs.com/talos/v1.13/deploy-and-manage-workloads/scaling-up
How to add more nodes to a Talos Linux cluster.
To add more nodes to a Talos Linux cluster, follow the same procedure as when initially creating the cluster:
* Boot the new machines to install Talos Linux
* Apply the appropriate configuration file (`worker.yaml` or `controlplane.yaml`) to each new machine
## Get the appropriate machine configuration
You need a valid `controlplane.yaml` or `worker.yaml` to apply to new nodes. If your cluster has been upgraded since it was initially created, your original files may be stale, regenerate them instead using the [reproducible machine configuration workflow](../configure-your-talos-cluster/system-configuration/reproducible-machine-configuration).
## Apply configuration to new nodes
Once you have the machine config, apply it to each new node using its IP address:
```bash theme={null}
talosctl apply-config --insecure \
--nodes \
--file
```
The `--insecure` flag is required because the PKI infrastructure has not yet been made available to the new node.
You do not need to bootstrap the new node. Whether you are adding a control plane or worker node, it will join the cluster automatically in its assigned role.
# Enable workers on your control plane nodes
Source: https://docs.siderolabs.com/talos/v1.13/deploy-and-manage-workloads/workers-on-controlplane
How to enable workers on your control plane nodes.
By default, Talos Linux taints control plane nodes so that workloads are not schedulable on them.
In order to allow workloads to run on the control plane nodes (useful for single node clusters, or non-production clusters), follow the procedure below.
Modify the machine configuration for the controlplane nodes to add `allowSchedulingOnControlPlanes: true`:
```yaml theme={null}
cluster:
allowSchedulingOnControlPlanes: true
```
This may be done via editing the `controlplane.yaml` file before it is applied to the control plane nodes, by [editing the machine config](../configure-your-talos-cluster/system-configuration/editing-machine-configuration), or by [patching the machine config](../configure-your-talos-cluster/system-configuration/patching).
## Load balancer configuration
When a load balancer such as MetalLB is used, the nodeLabel `node.kubernetes.io/exclude-from-external-load-balancers` should also be removed from the control plane nodes.
This label is added by default and instructs load balancers to exclude the node from the list of backend servers used by external load balancers.
In order to remove this label, you can patch the machine configuration for the control plane nodes with the patch:
```yaml theme={null}
machine:
nodeLabels:
node.kubernetes.io/exclude-from-external-load-balancers:
$patch: delete
```
# Deploy Your First Workload to a Talos Cluster
Source: https://docs.siderolabs.com/talos/v1.13/getting-started/deploy-first-workload
Deploy a sample workload to your Talos cluster to get started.
Deploying your first workload validates that your cluster is working properly and that you can schedule, expose, and access applications successfully.
If you don’t have a cluster running yet, check out the [Getting Started](./getting-started) or [Production Notes](./prodnotes) guides to learn how to create one.
To deploy a sample application to your cluster:
1. Run this command to deploy the application and expose it via a NodePort:
```bash theme={null}
kubectl apply -f https://raw.githubusercontent.com/siderolabs/example-workload/refs/heads/main/deploy/example-svc-nodeport.yaml
```
2. Verify that your pods and services are running:
```bash theme={null}
kubectl get pods,services # Lists the deployed pods and services
```
3. Create a `NODE_IP` variable by retrieving an IP address of any one of your nodes:
```bash theme={null}
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}'; echo)
```
4. Retrieve the NodePort assigned to your Service using:
```bash theme={null}
NODE_PORT=$(kubectl get svc example-workload -o jsonpath='{.spec.ports[0].nodePort}')
```
5. Verify your application is running:
```bash theme={null}
curl http://$NODE_IP:$NODE_PORT
```
And here is your application:
```text theme={null}
🎉 CONGRATULATIONS! 🎉
========================================
You successfully deployed the example workload!
Resources:
----------
🔗 Talos Linux: https://talos.dev
🔗 Omni: https://omni.siderolabs.com
🔗 Sidero Labs: https://siderolabs.com
========================================
```
## What’s next?
* [Pod Security](../../../kubernetes-guides/security/pod-security)
* [Set up persistent storage](../../../kubernetes-guides/csi/storage)
* [Deploy a Metrics Server](../../../kubernetes-guides/monitoring-and-observability/deploy-metrics-server)
* [Explore the talosctl CLI reference](../reference/cli)
# Getting Started
Source: https://docs.siderolabs.com/talos/v1.13/getting-started/getting-started
A guide to setting up a Talos cluster
This guide walks you through creating a simple Talos cluster with one control plane node and one or more worker nodes.
If you're looking to set up a cluster with multiple control plane nodes, see [Production Notes](./prodnotes).
**New to Talos?** Start with [Quickstart](./quickstart) to create a local virtual cluster on your workstation.
**Planning for production?** See [Production Notes](./prodnotes) for additional requirements and best practices.
**Installing on cloud or virtualized platforms?** Check out the [platform-specific guides](../platform-specific-installations) for installation methods tailored to different environments.
## Prerequisites
To create a Kubernetes cluster with Talos, you’ll need to:
* **Install talosctl**: `talosctl` is the CLI tool used to interact with the Talos API.
Since Talos Linux does not have SSH access, `talosctl` is the primary tool for managing and configuring your Talos machines
You can install `talosctl` on macOS or Linux by running:
```bash theme={null}
brew install siderolabs/tap/talosctl
```
Refer to the [talosctl installation guide](./talosctl) for installation on other platforms.
* **Ensure network access**: Your machines will need internet access to download the Talos installer and container images, sync time, and more.
If you’re working in a restricted network environment, check out the official documentation on using [registry proxies](../configure-your-talos-cluster/images-container-runtime/pull-through-cache), local registries, or setting up an [air-gapped installation](../platform-specific-installations/air-gapped).
## Talos cluster setup overview
Every Talos cluster follows the same process, regardless of where you deploy it:
1. **Boot** - Start machines with the Talos Linux image
2. **Configure** - Create a root of trust certificate authority and generate configuration files
3. **Apply** - Apply machine configurations to the nodes
4. **Connect** - Set up your local `talosctl` client
5. **Bootstrap** - Initialize the Kubernetes cluster.
**Note**: You can also use [Omni](https://www.siderolabs.com/omni-signup/) to create a Talos cluster that spans multiple platforms, including bare metal, cloud providers, and virtual machines.
Let's walk through each step and create a Talos cluster.
## Step 1: Download the Talos Linux image
Get the latest ISO for your architecture from our [Image factory](https://factory.talos.dev/).
## Step 2: Boot your machine
Boot your hardware using the ISO you just downloaded.
At this stage, you'll:
* Boot one machine as your control plane node.
* Boot additional machines as worker nodes (this is optional).
Once your hardware boots from the ISO image, you'll see the Talos dashboard. The node is now in maintenance mode, meaning it has no machine config yet and is waiting for one to be applied.
In maintenance mode, the Talos maintenance API is unauthenticated by design — anyone with network access can apply a machine config via `talosctl apply-config --insecure`. This is expected behavior. You can avoid publicly exposing the API in a number of ways:
* Apply a config at boot time by providing a full machine config via user-data, inline config, or embedded config.
* Configure node firewall rules to restrict which hosts can reach the maintenance API port, limiting exposure at the network level.
* Use a [Machine Join Config](../../../omni/omni-cluster-setup/registering-machines/join-machines-to-omni) to register the node with Omni, where the maintenance API is only accessible via SideroLink.
**Note:** The ISO runs entirely in RAM and won't modify your disks until you apply a configuration.
**Troubleshooting network connectivity:** If your machine fails to establish a network connection after booting, you may need to add network drivers through system extensions.
Add these extensions to your Talos image via the [Image factory](https://factory.talos.dev/), or see the [system extensions repository](https://github.com/siderolabs/extensions) for more information.
## Step 3: Store your node IP addresses in a variable
To create variables for your machines’ IP addresses:
1. Copy the IP address displayed on each machine console, including the control plane and any worker nodes you’ve created.
If you don’t have a display connected, retrieve the IP addresses from your DHCP server.
2. Create a variable for your control plane node’s IP address by replacing `` with the actual IP:
```bash theme={null}
export CONTROL_PLANE_IP=
```
3. If you have worker nodes, store their IP addresses in a Bash array.
Replace each `` placeholder with the actual IP address of a worker node.
You can include as many IP addresses as needed:
```bash theme={null}
WORKER_IP=("" "" ""...)
```
## Step 4: Unmount the ISO
Unplug your installation USB drive or unmount the ISO.
This prevents you from accidentally installing to the USB drive and makes it clearer which disk to select for installation.
## Step 5: Learn about your installation disks
When you first boot your machine from the ISO, Talos runs temporarily in memory.
This means that your Talos nodes, configurations, and cluster membership won't survive reboots or power cycles.
However, once you apply the machine configuration (which you'll do later in this guide), you'll install Talos, its complete operating system, and your configuration to a specified disk for permanent storage.
Run this command to view all the available disks on your control plane:
```bash theme={null}
talosctl get disks --insecure --nodes $CONTROL_PLANE_IP
```
Note the disk ID (e.g., `sda`, `vda`) as you will use it in the next step.
## Step 6: Generate cluster configuration
Talos Linux is configured entirely using declarative configuration files avoiding the need to deal with SSH and running commands.
To generate these declarative configuration files:
1. Define variables for your cluster name and the disk ID from step 5.
Replace the placeholders with your actual values:
```bash theme={null}
export CLUSTER_NAME=
export DISK_NAME=
```
2. Run this command to generate the configuration file:
```bash theme={null}
talosctl gen config $CLUSTER_NAME https://$CONTROL_PLANE_IP:6443 --install-disk /dev/$DISK_NAME
```
This command generates machine configurations that specify the Kubernetes API endpoint (which is your control plane node's IP) for cluster communication and the target disk for the Talos installation.
You'll get three files from this command:
* **controlplane.yaml**: The configuration for your control plane.
* **worker.yaml**: The configuration for your worker nodes.
* **talosconfig**: Your `talosctl` configuration file, used to connect to and authenticate access to your cluster.
## Step 7: Apply configurations
Now that you've created your configurations, it's time to apply them to bring your nodes and cluster online:
1. Run this command to apply the control plane configuration:
```bash theme={null}
talosctl apply-config --insecure --nodes $CONTROL_PLANE_IP --file controlplane.yaml
```
2. Next, apply the worker node configuration:
```bash theme={null}
for ip in "${WORKER_IP[@]}"; do
echo "Applying config to worker node: $ip"
talosctl apply-config --insecure --nodes "$ip" --file worker.yaml
done
```
## Step 8: Set your endpoints
An endpoint is a control plane node or [Omni](../../../omni/getting-started/getting-started) domain. You can read more about the architecture [here](../learn-more/talosctl).
Set your endpoints with this:
```bash theme={null}
talosctl --talosconfig=./talosconfig config endpoints $CONTROL_PLANE_IP
```
## Step 9: Bootstrap your etcd cluster
Wait for your control plane node to finish booting, then bootstrap your etcd cluster by running:
```bash theme={null}
talosctl bootstrap --nodes $CONTROL_PLANE_IP --talosconfig=./talosconfig
```
**Note:** Run this command ONCE on a SINGLE control plane node.
If you have multiple control plane nodes, you can choose any of them.
## Step 10: Get Kubernetes access
Download your `kubeconfig` file to start using `kubectl`.
You have two download options: you can either merge your Kubernetes configurations **OR** keep them separate.
Here’s how to do both:
* Merge your new cluster into your local Kubernetes configuration:
```bash theme={null}
talosctl kubeconfig --nodes $CONTROL_PLANE_IP --talosconfig=./talosconfig
```
* Specify a filename if you prefer not to merge with your default Kubernetes configuration:
```bash theme={null}
talosctl kubeconfig alternative-kubeconfig --nodes $CONTROL_PLANE_IP --talosconfig=./talosconfig
export KUBECONFIG=./alternative-kubeconfig
```
## Step 11: Check cluster health
Run the following command to check the health of your nodes:
```bash theme={null}
talosctl --nodes $CONTROL_PLANE_IP --talosconfig=./talosconfig health
```
## Step 12: Verify node registration
Confirm that your nodes are registered in Kubernetes:
```bash theme={null}
kubectl get nodes
```
You should see your control plane and worker nodes listed with a **Ready** status.
## Next steps
Congratulations!
You now have a working Kubernetes cluster on Talos Linux.
For a list of all the commands and operations that `talosctl` provides, see the CLI reference.
### What's next?
* [Set up persistent storage](../../../kubernetes-guides/csi/storage)
* [Deploy a Metrics Server](../../../kubernetes-guides/monitoring-and-observability/deploy-metrics-server)
* [Explore the talosctl CLI reference](../reference/cli)
* [Plan your production deployment](./prodnotes)
# Production Clusters
Source: https://docs.siderolabs.com/talos/v1.13/getting-started/prodnotes
Recommendations for setting up a Talos Linux cluster in production.
This guide explains things to consider to create a production quality Talos Linux cluster for bare metal.
Check out the [Reference Architecture documentation](https://www.siderolabs.com/resources/) for architectural diagrams and guidance on creating production-grade clusters in other environments.
This guide assumes that you’ve already created a development cluster and are familiar with the **Getting Started** documentation.
If not, please refer to the [Getting Started](./getting-started) guide for more information.
When moving from a learning environment to a production-ready Talos Linux cluster, you have to consider several critical factors:
* High availability for your control plane nodes.
* Secure configuration management.
* Reliability for continuous service and minimal downtime.
* Authentication for access control.
Follow the steps below to build a production-grade Talos cluster that is highly available, reliable, and secure.
**Note**: Check out [Omni](https://www.siderolabs.com/omni-signup/) for managing large-scale Talos Linux clusters automatically.
## Step 1: Prepare your infrastructure
To create your production cluster infrastructure:
1. Boot your machines using the Talos ISO image
2. Ensure network access on your nodes.
Here is how to do each step:
### Boot your machines using the Talos ISO image
Follow these steps to boot your machines using the Talos ISO image:
1. Download the latest ISO for your infrastructure depending on the hardware type from the [Talos Image factory](https://factory.talos.dev/).
**Note**: For network booting and self-built media using published kernel there are a number of required kernel parameters.
Please see the [kernel docs](../reference/kernel) for more information.
2. Boot three control planes using the ISO image you just downloaded.
3. Boot additional machines as worker nodes.
### Ensure network access
If your nodes are behind a firewall, in a private network, or otherwise not directly reachable, you would need to configure a load balancer to forward TCP port 50000 to reach the nodes for Talos API access.
**Note**: Because the Talos Linux API uses gRPC and mutual TLS, it cannot be proxied by a HTTP/S proxy, but only by a TCP load balancer.
With your control plane and worker nodes booted, next configure your Kubernetes endpoint.
## Step 2: Store your IP addresses in a variable
To store variables for your machines’ IP addresses:
1. Copy the IP address displayed on each machine console, including the control plane and any worker nodes you’ve created.
If you don’t have a display connected, retrieve the IP addresses from your DHCP server.
2. Create a Bash array for your control plane node IP addresses, replacing each `` placeholder with the IP address of a control plane node.
You can include as many IP addresses as needed:
```bash theme={null}
CONTROL_PLANE_IP=("" "" "")
```
**For example**:
If your control plane nodes IP addresses are `192.168.0.2`, `192.168.0.3`, `192.168.0.4`, your command would be:
```bash theme={null}
CONTROL_PLANE_IP=("192.168.0.2" "192.168.0.3" "192.168.0.4")
```
3. If you have worker nodes, store their IP addresses in a Bash array.
Replace each `` placeholder with the actual IP address of a worker node.
You can include as many IP addresses as needed:
```bash theme={null}
WORKER_IP=("" "" "")
```
## Step 3: Decide your Kubernetes endpoint
You've set up multiple control planes for high availability, but they only provide true high availability if the Kubernetes API server endpoint can reach all control plane nodes.
Here are two common ways to configure this:
* **Dedicated load balancer**: Set a dedicated load balancer that routes to your control plane nodes.
* **DNS records**: Create multiple DNS records that point to all your control plane nodes
With these, you can pass in one IP address or DNS name during setup that route to all your control plane nodes.
Here is how you can configure each option:
### Dedicated load balancer
If you're using a cloud provider or have your own load balancer (such as HAProxy, an NGINX reverse proxy, or an F5 load balancer), setting up a dedicated load balancer is a natural choice.
It is also important to note that if you [created the cluster with Omni](https://omni.siderolabs.com/tutorials/getting_started), Omni will automatically be a load balancer for your Kubernetes endpoint.
Configure a frontend to listen on TCP port 6443 and direct traffic to the addresses of your Talos control plane nodes.
Your Kubernetes endpoint will be the IP address or DNS name of the load balancer's frontend, with the port appended, for example, `https://myK8s.mydomain.io:6443`.
**Note**: You cannot use a HTTP load balancer, because the Kubernetes API server handles TLS termination and mutual TLS authentication.
### DNS records
Additionally, you can configure your Kubernetes endpoint using DNS records.
Simply, add multiple A or AAAA records, one for each control plane, to a DNS name.
For example, you can add:
```url theme={null}
kube.cluster1.mydomain.com IN A 192.168.0.10
kube.cluster1.mydomain.com IN A 192.168.0.11
kube.cluster1.mydomain.com IN A 192.168.0.12
```
Then your endpoint would be:
```url theme={null}
https://kube.cluster1.mydomain.com:6443
```
> **Note**: This endpoint serves as a fallback if [KubePrism](../../../kubernetes-guides/advanced-guides/kubeprism) becomes unavailable, for example, after a reboot. It also ensures that Talos nodes can still locate and communicate with the control plane if other discovery services are not available.
## Step 4: Save your endpoint in a variable
Set a variable to store the endpoint you chose in Step 3.
Replace `` placeholder with your actual endpoint:
```bash theme={null}
export YOUR_ENDPOINT=
```
## Step 5: Generate secrets bundle
The secrets bundle is a file that contains all the cryptographic keys, certificates, and tokens needed to secure your Talos Linux cluster.
To generate the secrets bundle, run:
```bash theme={null}
talosctl gen secrets -o secrets.yaml
```
## Step 6: Generate machine configurations
Follow these steps to generate machine configuration:
1. Set a variable for your cluster name by running the following command.
Replace `` with the name you want to give your cluster:
```bash theme={null}
export CLUSTER_NAME=
```
2. Run this command to generate your machine configuration files using your secrets bundle:
```bash theme={null}
talosctl gen config --with-secrets secrets.yaml $CLUSTER_NAME https://$YOUR_ENDPOINT:6443
```
The `talosctl` version (`talosctl version --client`) should match the Talos OS version installed on your nodes.
If you are using a newer version of `talosctl` to generate configurations for an older Talos OS, use the `--talos-version` flag to ensure compatibility.
For example, to generate a configuration compatible with Talos v1.13:
```bash theme={null}
talosctl gen config --with-secrets secrets.yaml --talos-version v1.13 $CLUSTER_NAME https://$YOUR_ENDPOINT:6443
```
Even when using `--talos-version`, review the generated configuration files to verify that the Kubernetes version is supported by your target Talos OS.
Refer to the [support matrix](./support-matrix) to check which Kubernetes versions are compatible with your Talos version.
This command will generate three files:
* **controlplane.yaml**: Configuration for your control plane.
* **worker.yaml**: Configuration for your worker nodes.
* **talosconfig**: The `talosctl` configuration file used to connect to and authenticate with your cluster.
## Step 7: Unmount the ISO
Unplug your installation USB drive or unmount the ISO from all your control plane and worker nodes.
This prevents you from accidentally installing to the USB drive and makes it clearer which disk to select for installation.
## Step 8: Understand your nodes
The default machine configurations for control plane and worker nodes are typically sufficient to get your cluster running.
However, you may need to customize certain settings such as network interfaces and disk configurations depending on your specific environment.
Follow these steps to verify that your machine configurations are set up correctly:
1. **Check network interfaces**: Run this command to view all network interfaces on any node, whether control plane or worker.
Replace `` with the IP of the node you want to inspect.
**Note**: Copy the network ID with an Operational state (OPER) value of **up**.
```bash theme={null}
talosctl --nodes get links --insecure
```
2. **Check Available Disks:** Run this command to check all available disks on any node.
Replace `` with the IP address of the node you want to inspect:
```bash theme={null}
talosctl get disks --insecure --nodes
```
3. **Verify Configuration Files:** Open your `worker.yaml` and `controlplane.yaml` configuration files in your preferred editor.
Check that the values match your worker and control plane node's network and disk settings.
If the values don't match, you'll need to update your machine configuration.
**Note**: Refer to the [Talos CLI reference](../reference/cli) for additional commands to gather more information about your nodes and cluster.
## Step 9: Patch your machine configuration (optional)
You can patch your worker and control plane machine configuration to reflect the correct network interface and disk of your control plane nodes.
Follow these steps to patch your machine configuration:
1. Create patch files for the configurations you want to modify:
```bash theme={null}
touch controlplane-patch-1.yaml # For patching the control plane nodes configuration
touch worker-patch-1.yaml # For patching the worker nodes configuration
```
**Note**: You don't have to create both patch files, only create patches for the configurations you actually need to modify.
You can also create multiple patch files (e.g., `controlplane-patch-2.yaml`, `controlplane-patch-3.yaml`) if you want to make multiple subsequent patches to the same machine configuration.
2. Copy and paste this YAML block of code and add the correct hardware values to each patch file.
For example, for `controlplane-patch-1` use the network interface and disk information you gathered from your control plane nodes:
```yaml theme={null}
# controlplane-patch-1 file
machine:
network:
interfaces:
- interface: # From control plane node
dhcp: true
install:
disk: /dev/ # From control plane node
```
For `worker-patch-1.yaml`, use network interface and disk information from your worker nodes:
```yaml theme={null}
# worker-patch-1.yaml file
machine:
network:
interfaces:
- interface: # From worker node
dhcp: true
install:
disk: /dev/ # From worker node
```
3. Apply the different patch files for the different machine configuration:
* **For control plane**:
```bash theme={null}
talosctl machineconfig patch controlplane.yaml --patch @controlplane-patch-1.yaml --output controlplane.yaml
```
* **For worker**:
```bash theme={null}
talosctl machineconfig patch worker.yaml --patch @worker-patch-1.yaml --output worker.yaml
```
Additionally, you can learn more about [patches](../configure-your-talos-cluster/system-configuration/patching) from the configuration patches documentation.
## Step 10: Configure your multihomed machines
If your machines are multihomed, i.e., they have more than one IPv4 and/or IPv6 addresses other than loopback, then additional configuration is required.
Refer to [Multihoming](../networking/multihoming) for more information.
## Step 11: Apply the machine configuration
To apply your machine configuration:
1. Run this command to apply the `controlplane.yaml` configuration to your control plane nodes:
```bash theme={null}
for ip in "${CONTROL_PLANE_IP[@]}"; do
echo "=== Applying configuration to node $ip ==="
talosctl apply-config --insecure \
--nodes $ip \
--file controlplane.yaml
echo "Configuration applied to $ip"
echo ""
done
```
2. Run this command to apply the `worker.yaml`configuration to your worker node:
```bash theme={null}
for ip in "${WORKER_IP[@]}"; do
echo "=== Applying configuration to node $ip ==="
talosctl apply-config --insecure \
--nodes $ip \
--file worker.yaml
echo "Configuration applied to $ip"
echo ""
done
```
## Step 12: Manage your Talos configuration file
The `talosconfig` is your key to managing the Talos Linux cluster, without it, you cannot authenticate or communicate with your cluster nodes using `talosctl`.
You have two options for managing your `talosconfig`:
1. Merge your new `talosconfig` into the default configuration file located at `~/.talos/config`:
```bash theme={null}
talosctl config merge ./talosconfig
```
2. Copy the configuration file to your `~/.talos` directory and set the `TALOSCONFIG` environment variable:
```bash theme={null}
mkdir -p ~/.talos
cp ./talosconfig ~/.talos/config
export TALOSCONFIG=~/.talos/config
```
## Step 13: Set endpoints of your control plane nodes
Configure your endpoints to enable talosctl to automatically load balance requests and fail over between control plane nodes when individual nodes become unavailable.
Run this command to configure your endpoints.
Replace the placeholders ` ` with the IP addresses of your control plane nodes:
```bash theme={null}
talosctl config endpoint
```
**For example**:
If your control plane nodes IP addresses are `192.168.0.2`, `192.168.0.3`, `192.168.0.4`, your command would be:
```bash theme={null}
talosctl config endpoint 192.168.0.2 192.168.0.3 192.168.0.4
```
## Step 14: Bootstrap your Kubernetes cluster
Wait for your control plane nodes to finish booting, then bootstrap your etcd cluster by running the command below.
Replace the `` placeholder with the IP address of ONE of your three control plane nodes:
```bash theme={null}
talosctl bootstrap --nodes
```
**Note**: Run this command ONCE on a SINGLE control plane node.
If you have multiple control plane nodes, you can choose any of them.
## Step 15: Get Kubernetes access
Download your `kubeconfig` file to start using `kubectl` with your cluster.
These commands must be run against a single control plane node.
You have two options for managing your `kubeconfig`.
Replace `` with the IP address of any one of your control plane nodes:
* Merge into your default `kubeconfig`:
```bash theme={null}
talosctl kubeconfig --nodes
```
* Create a separate `kubeconfig` file:
```bash theme={null}
talosctl kubeconfig alternative-kubeconfig --nodes
export KUBECONFIG=./alternative-kubeconfig
```
## Step 16: Verify your nodes are running
Run the command to ensure that your nodes are running:
```bash theme={null}
kubectl get nodes
```
## Next steps
Congratulations!
You now have a working production grade Talos Linux Kubernetes cluster.
### What's next?
* [Set up persistent storage](../../../kubernetes-guides/csi/storage)
* [Deploy a Metrics Server](../../../kubernetes-guides/monitoring-and-observability/deploy-metrics-server)
* [Explore the talosctl CLI reference](../reference/cli)
# Quickstart
Source: https://docs.siderolabs.com/talos/v1.13/getting-started/quickstart
A short guide on setting up a simple Talos Linux cluster locally with Docker.
The easiest way to run Talos is by using the CLI (`talosctl`) to create a cluster on a machine with `docker` installed.
### Prerequisites
#### `talosctl`
Download `talosctl` (macOS or Linux):
```bash theme={null}
brew install siderolabs/tap/talosctl
```
#### `kubectl`
Download `kubectl` via one of the methods outlined in the [documentation](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
### Create the cluster
Now run the following:
```bash theme={null}
talosctl cluster create docker
```
If you are using Docker Desktop on a macOS computer, if you encounter the error: *Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?* you may need to manually create the link for the Docker socket:
`sudo ln -s "$HOME/.docker/run/docker.sock" /var/run/docker.sock`
You can explore using Talos API commands:
```bash theme={null}
talosctl dashboard --nodes 10.5.0.2
```
Verify that you can reach Kubernetes:
### Destroy the cluster
When you are all done, remove the cluster:
```bash theme={null}
talosctl cluster destroy
```
# Support Matrix
Source: https://docs.siderolabs.com/talos/v1.13/getting-started/support-matrix
Table of supported Talos Linux versions and respective platforms.
| Talos Version | 1.13 | 1.12 |
| ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Release Date | 2026-04-27 (TBD) | 2025-12-22 (1.12.0) |
| End of Community Support | 1.14.0 release (2026-08-30, TBD) | 1.13.0 release (2026-04-27, TBD) |
| Enterprise Support | [offered by Sidero Labs Inc.](https://www.siderolabs.com/support/) | [offered by Sidero Labs Inc.](https://www.siderolabs.com/support/) |
| Kubernetes | 1.36, 1.35, 1.34, 1.33, 1.32, 1.31 | 1.35, 1.34, 1.33, 1.32, 1.31, 1.30 |
| NVIDIA Drivers | 595.x.x (PRODUCTION), 580.x.x (LTS) | 570.x.x (PRODUCTION), 580.x.x (LTS) |
| Architecture | amd64, arm64 | amd64, arm64 |
| **Platforms** | | |
| - cloud | Akamai, AWS, GCP, Azure, CloudStack, Digital Ocean, Exoscale, Hetzner, OpenNebula, OpenStack, Oracle Cloud, Scaleway, Vultr, Upcloud | Akamai, AWS, GCP, Azure, CloudStack, Digital Ocean, Exoscale, Hetzner, OpenNebula, OpenStack, Oracle Cloud, Scaleway, Vultr, Upcloud |
| - bare metal | x86: BIOS, UEFI, SecureBoot; arm64: UEFI, SecureBoot; boot: ISO, PXE, disk image | x86: BIOS, UEFI; arm64: UEFI; boot: ISO, PXE, disk image |
| - virtualized | VMware, Hyper-V, KVM, Proxmox, Xen | VMware, Hyper-V, KVM, Proxmox, Xen |
| - SBCs | Banana Pi M64, Jetson Nano, Libre Computer Board ALL-H3-CC, Nano Pi R4S, Pine64, Pine64 Rock64, Radxa ROCK Pi 4C, Radxa ROCK 4C+, Radxa ROCK 5B, Raspberry Pi 4B, Raspberry Pi Compute Module 4, Turing RK1, Orange Pi 5 | Banana Pi M64, Jetson Nano, Libre Computer Board ALL-H3-CC, Nano Pi R4S, Pine64, Pine64 Rock64, Radxa ROCK Pi 4C, Radxa ROCK 4C+, Radxa ROCK 5B, Raspberry Pi 4B, Raspberry Pi Compute Module 4, Turing RK1, Orange Pi 5 |
| - local | Docker, QEMU | Docker, QEMU |
| **Omni** | | |
| [Omni](https://github.com/siderolabs/omni) | >= 1.8.0 | >= 1.4.0 |
| **Cluster API** | | |
| [CAPI Bootstrap Provider Talos](https://github.com/siderolabs/cluster-api-bootstrap-provider-talos) | >= 0.6.12 | >= 0.6.11 |
| [CAPI Control Plane Provider Talos](https://github.com/siderolabs/cluster-api-control-plane-provider-talos) | >= 0.5.13 | >= 0.5.12 |
| [Sidero](https://www.sidero.dev/) | >= 0.6.13 | >= 0.6.12 |
## Platform tiers
* Tier 1: Automated tests, high-priority fixes.
* Tier 2: Tested from time to time, medium-priority bugfixes.
* Tier 3: Not tested by core Talos team, community tested.
### Tier 1
* Metal
* AWS
* GCP
### Tier 2
* Azure
* Digital Ocean
* OpenStack
* VMWare
### Tier 3
* Akamai
* CloudStack
* Exoscale
* Hetzner
* nocloud
* OpenNebula
* Oracle Cloud
* Scaleway
* Vultr
* Upcloud
# System Requirements
Source: https://docs.siderolabs.com/talos/v1.13/getting-started/system-requirements
Hardware requirements for running Talos Linux.
## Minimum requirements
| Role |
Memory |
Cores |
System Disk |
| Control Plane |
2 GiB |
2 |
10 GiB |
| Worker |
1 GiB |
1 |
10 GiB |
## Recommended
| Role |
Memory |
Cores |
System Disk |
| Control Plane |
4 GiB |
4 |
100 GiB |
| Worker |
2 GiB |
2 |
100 GiB |
These requirements are similar to that of Kubernetes.
## Storage
Talos Linux itself only requires less than 100 MB of disk space, but the EPHEMERAL partition is used to store pulled images, container work directories, and so on. Thus a minimum of 10 GiB of disk space is required. 100 GiB is recommended.
Talos manages disk partitioning automatically during installation, creating EFI, META, STATE, and EPHEMERAL partitions. The EPHEMERAL partition then expands to fill all the space left after the first three. That space can either remain entirely with EPHEMERAL or be divided into additional user volumes, depending on your needs. See [Disk Layout](../configure-your-talos-cluster/storage-and-disk-management/disk-management#disk-layout) for details.
For production, it is often more efficient to dedicate a smaller disk for the Talos installation itself, and use additional disks for workload storage. Using a large, single disk for both system and workloads is supported, but may not be optimal depending on your environment.
# talosctl
Source: https://docs.siderolabs.com/talos/v1.13/getting-started/talosctl
Install Talos Linux CLI
## Recommended
The client can be installed and updated via the [Homebrew package manager](https://brew.sh/) for macOS and Linux.
You will need to install `brew` and then you can install `talosctl` from the Sidero Labs tap.
```bash theme={null}
brew install siderolabs/tap/talosctl
```
This will also keep your version of `talosctl` up to date with new releases.
This homebrew tap also has formulae for `omnictl` if you need to install that package.
> Note: Your `talosctl` version should match the version of Talos Linux you are running on a host.
> To install a specific version of `talosctl` with `brew` you can follow [this github issue](https://github.com/siderolabs/homebrew-tap/issues/75).
## Alternative install
You can automatically install the correct version of `talosctl` for your operating system and architecture with an installer script.
This script won't keep your version updated with releases and you will need to re-run the script to download a new version.
```bash theme={null}
curl -sL https://talos.dev/install | sh
```
This script will work on macOS, Linux, and WSL on Windows.
It supports amd64 and arm64 architecture.
## Manual and Windows install
All versions can be manually downloaded from the [talos releases page](https://github.com/siderolabs/talos/releases/) including Linux, macOS, and Windows.
You will need to add the binary to a folder part of your executable `$PATH` to use it without providing the full path to the executable.
Updating the binary will be a manual process.
# What's New in Talos 1.13.0
Source: https://docs.siderolabs.com/talos/v1.13/getting-started/what's-new-in-talos
Discover the latest features and updates in Talos Linux 1.13.
For critical changes, refer to the [upgrade notes](../configure-your-talos-cluster/lifecycle-management/upgrading-talos).
## Important changes
### `talosctl debug`
Talos Linux now provides a way to run and attach to the [privileged debug container](../troubleshooting/talosctl-debug) with a user-provided container image.
The debug container might be used for troubleshooting and debugging purposes.
### Container Image Signature Verification
Talos now supports machine-wide container [image signature verification](../security/verifying-image-signatures) via the new [`ImageVerificationConfig`](../reference/configuration/security/imageverificationconfig) machine config document.
Any image which gets pulled on the node will be verified against the configured rules, and if no rule matches, it will be pulled without verification.
### NVIDIA GPU Support
Talos switched to using CDI and now supports configuring NVIDIA GPU via the gpu-operator helm chart.
See the [upgrade notes](../configure-your-talos-cluster/lifecycle-management/upgrading-talos#nvidia-gpu-users)
for more details on how to configure NVIDIA GPU support in Talos.
### Flannel CNI with Network Policy Support
Talos Linux now supports optionally deploying Flannel CNI with [network policy support](https://kubernetes.io/docs/concepts/services-networking/network-policies/) enabled.
The network policy implementation is [kube-network-policies](https://github.com/kubernetes-sigs/kube-network-policies/).
To enable Flannel CNI with network policy support, use the following machine configuration patch:
```yaml theme={null}
cluster:
network:
cni:
name: flannel
flannel:
kubeNetworkPoliciesEnabled: true
```
(If the cluster is already running, sync the bootstrap manifests after applying the patch to deploy the new CNI configuration.)
### Kubernetes Bootstrap Manifests
Talos now uses inventory-backed [server-side apply](https://kubernetes.io/docs/reference/using-api/server-side-apply/) when applying bootstrap manifests
(including `extraManifests` and `inlineManifests`).
Purging of unneeded manifests is automatically performed by [`talosctl upgrade-k8s`](../../../kubernetes-guides/advanced-guides/upgrading-kubernetes/).
The switch and inventory backfill is automatic and no action is needed from the user.
### Upgrade Flow
Talos now exposes install and upgrade operations via the `LifecycleService` API, enabling programmatic installs and upgrades through a single, consistent interface.
The legacy upgrade API is deprecated; new integrations should migrate to `LifecycleService` for future compatibility.
`talosctl` upgrades now route through `LifecycleService`, aligning CLI behavior with the new install/upgrade API and unifying the upgrade path.
This change is transparent to users but standardizes the backend used for upgrades.
A user-facing change is that upgrade process (running new boot assets) happens while the machine is still running workloads, followed by a machine reboot wrapped with
Kubernetes node drain and uncordon. Draining the node is now optional.
## Storage Subsystem
### External Volumes
Talos now supports virtiofs-based [external volumes](../configure-your-talos-cluster/storage-and-disk-management/disk-management/external) via the new
[ExternalVolumeConfig](../reference/configuration/block/externalvolumeconfig/)
document.
The virtiofs external volumes are not supported when SELinux is running in [enforcing mode](../security/selinux).
### Negative Max Volume Size
Negative max size represents the amount of space to be left free on the device, rather than the size the volume should consume.
For example:
* a max size of "-10GiB" means the volume can grow to the available space minus 10GiB.
* a max size of "-25%" means the volume can grow to the available space minus 25%.
## Network Subsystem
### KubeSpan Configuration
A new `KubeSpanConfig` document has been introduced to configure KubeSpan settings.
It replaces and deprecates the previous method of configuring KubeSpan via the `.machine.network.kubespan` field.
The old configuration field will continue to work for backward compatibility.
KubeSpan now supports filtering of advertised networks using the `excludeAdvertisedNetworks` field in the `KubeSpanConfig` document.
This allows users to specify a list of CIDRs to exclude from the advertised networks. Please note that routing must be symmetric for any
pair of peers, so if one peer excludes a certain network, the other peer must also exclude it. In other words, for any given pair of peers,
and any pair of their addresses, the traffic should either go through KubeSpan or not, but not one way or the other.
### LinkAliasConfig Pattern-Based Multi-Alias
[`LinkAliasConfig`](../reference/configuration/network/linkaliasconfig/) now supports pattern-based alias names using `%d` format verb (e.g. `net%d`).
When the alias name contains a `%d` format verb, the selector is allowed to match multiple links.
Each matched link receives a sequential alias (e.g. `net0`, `net1`, ...) based on hardware address order
of the links. Links already aliased by a previous config are automatically skipped.
This enables creating stable aliases from any N links using a single config document,
useful for `BondConfig` and `BridgeConfig` member interfaces on varying hardware.
### ProbeConfig
The [`TCPProbeConfig`](../reference/configuration/network/tcpprobeconfig/) configuration document allows to configure TCP probes for network reachability checks.
This allows to define a custom connectivity condition instead of Talos default condition (whether a default gateway exists).
### ResolverConfig
The nameservers configuration in [machine configuration](../reference/configuration/network/resolverconfig/) now overwrites any previous layers (defaults, platform, etc.) when specified.
Previously a smart merge was performed to keep IPv4/IPv6 nameservers from lower layers if the machine configuration specified only one type.
### Routing Rules Support
Talos now supports configuring Linux routing rules via the new [`RoutingRuleConfig`](../reference/configuration/network/routingruleconfig/) machine config document.
### VRF Support
Talos now supports VRF (Virtual Routing and Forwarding) via the new [`VRFConfig`](../reference/configuration/network/vrfconfig/) machine config document.
## Kubernetes
### Component Configuration Extra Arguments
Several Talos configuration fields that previously accepted single string values for extra arguments have been updated to accept slices of strings as well.
This includes fields such as `.cluster.apiServer.extraArgs`.
> Note: If you were relying on the resources EtcdConfigs, KubeletConfigs, ControllerManagerConfigs, SchedulerConfigs or APIServerConfigs, the protobuf format has changed from `map` to `map`.
## Miscellaneous
### Linux Kernel built with ThinLTO
Talos now uses a kernel built using `Clang` compiler, and optimized using ThinLTO. This should bring a small performance improvement,
alongside some hardening features, such as BTI on supported ARM systems.
### Dynamic Linux Kernel Preemption Model
Talos Linux now defaults to dynamic Linux kernel preemption model, the default value `none` matches
previous version, but now with kernel argument `preempt=` the preemption model can be changed.
See [Linux kernel documentation](https://docs.kernel.org/admin-guide/kernel-parameters.html) for more
information on supported values.
This change only applies to amd64 (x86\_64) architecture.
### Container Image Decompression
Talos now ships with `igzip` (amd64) and `pigz` (arm64) to speed up container image decompression.
### `/proc/PID/mem` Access Hardening
A new kernel parameter `proc_mem.force_override=never` has been introduced by default to enhance system security
by preventing unwanted writes to protected process memory via `/proc/PID/mem`.
If the kernel parameter is removed, default behavior is restored, allowing access only if the process is traced.
### Container Device Interface
Talos now enables [CDI](https://github.com/cncf-tags/container-device-interface) by default.
System extensions can bring in dynamic CDI spec files under `/run/cdi`.
### Resource Viewer in Interactive Dashboard
The [interactive dashboard](../deploy-and-manage-workloads/interactive-dashboard/) now includes a resource viewer screen, which provides a way to inspect the running Talos machine state when using the video console is the only option
(e.g. when the network is not working and `talosctl` cannot be used).
### VM Hot-Add Support
Talos now includes udev rules to support hot-adding of CPUs in virtualized environments.
### Talos Imager Enhancements
Talos [`imager`](../platform-specific-installations/boot-assets/#imager) now can run rootless.
The flags `--privileged` and `-v /dev:/dev` are no longer required for `docker run`.
Talos disk images are now reproducible. Building the same version of Talos multiple times will yield
identical disk images.
Note: VHD and VMDK (Azure and VMware) images are not currently reproducible due to limitations in the underlying image creation tools.
Users verifying reproducible images should use raw images, verify checksums, and convert them to VHD/VMDK as needed.
### Image APIs Updated
Talos Linux provides new APIs to manage container images on the node: listing, pulling, importing and removing images.
The new pull APIs provide pull progress notifications.
The CLI commands `talosctl image pull`, `talosctl image list` and `talosctl image remove` have been updated to interact with the new APIs.
### Environment Configuration Document
A new [`EnvironmentConfig`](../reference/configuration/runtime/environmentconfig) document has been introduced to allow users to specify environment variables for Talos components.
It replaces and deprecates the previous method of setting environment variables via the `.machine.env` field.
Multiple values for the same environment variable will replace previous values, with the last one taking precedence.
To remove an environment variable, remove it from the `EnvironmentConfig` document and restart the node.
### Image Bundles
The `talosctl images k8s-bundle` command now accepts an optional version override argument.
The `talosctl images talos-bundle` command now accepts optional `--overlays` and `--extensions` flags.
If those are set to `false`, the command will not attempt to reach out to the container registry to fetch the latest versions and digests of the overlays and extensions.
## Component Updates
* Linux: 6.18.24
* containerd: 2.2.3
* runc: 1.4.2
* etcd: 3.6.9
* CoreDNS: 1.14.2
* Kubernetes: 1.36.0
* CNI: 1.9.1
* Flannel CNI plugin: v1.9.1-flannel1
* Flannel: 0.28.2
* LVM2: 2\_03\_38
* systemd: 259.5
* cryptsetup: 2.8.3
* iptables: 1.8.12
* musl: 1.2.6
Talos is built with Go 1.26.2.
## Contributors
* Andrey Smirnov
* Mateusz Urbanek
* Noel Georgi
* Orzelius
* Mickaël Canévet
* Dmitrii Sharshakov
* Laura Brehm
* Artem Chernyshev
* Edward Sammut Alessi
* Fritz Schaal
* Max Makarov
* Andreas Freund
* Bryan Lee
* Justin Garrison
* Nico Berlee
* Pranav Patil
* Spencer Smith
* Utku Ozdemir
* Zadkiel AHARONIAN
* Alexis La Goutte
* Andras BALI
* Andreas Lüdeke
* Andrei Kvapil
* Birger Johan Nordølum
* Camillo Rossi
* Christopher Puschmann
* Daniil Kivenko
* David Orman
* Dmitrii Sharshakov
* Dominik Pitz
* Florian Ströger
* Gregor Gruener
* Jaakko Sirén
* Jan Paul
* Jean-Francois Roy
* Joakim Nohlgård
* Jonas Lammler
* Kai Zhang
* Kevin Tijssen
* Lennard Klein
* Matthew Sanabria
* Michal Baumgartner
* Olav Thoresen
* Serge van Ginderachter
* Skye Soss
* Stanley Chan
* Sébastien Masset
* Tim Jones
* arita
* dataprolet
* drew
* eseiker
* greenpsi
* lmacka
* pranav767
* pythoner6
# Blackhole Routes
Source: https://docs.siderolabs.com/talos/v1.13/networking/advanced/blackhole
How to configure blackhole routes in your network.
Blackhole routes are used to discard traffic destined for specific IP addresses or subnets.
It might be useful to set up blackhole routes for security reasons, to prevent traffic from reaching certain destinations,
or for temporary testing and troubleshooting purposes.
Blackhole routes can be configured in Talos Linux using the [`BlackholeRouteConfig`](../../reference/configuration/network/blackholerouteconfig) machine configuration document:
```yaml theme={null}
apiVersion: v1alpha1
kind: BlackholeRouteConfig
name: 169.254.1.1/32
```
This configuration creates a blackhole route for the IP address `169.254.1.1` - any traffic destined for this IP address will be discarded by the system.
Blackhole route will appear as a route attached to the loopback interface `lo` (for IPv6) or as a route attached to no interface (for IPv4) with the `blackhole` type:
```bash theme={null}
$ talosctl get routes
NODE NAMESPACE TYPE ID VERSION DESTINATION GATEWAY LINK METRIC
172.20.0.5 network RouteStatus inet4//169.254.1.1/32/1024 1 169.254.1.1/32 1024
```
# Ethernet Configuration
Source: https://docs.siderolabs.com/talos/v1.13/networking/advanced/ethernet-config
How to configure Ethernet network link settings.
Talos Linux allows you to configure Ethernet network link settings, such as ring configuration or disabling TCP checksum offloading.
The settings and their values closely follow `ethtool` command line options, so you can use similar recipes.
## Observing current status
You can observe current Ethernet settings in the `EthernetStatus` resource:
```yaml theme={null}
# talosctl get ethernetstatus enp0s2 -o yaml
spec:
rings:
rx-max: 256
tx-max: 256
rx: 256
tx: 256
tx-push: false
rx-push: false
features:
tx-scatter-gather: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
highdma: on [fixed]
tx-scatter-gather-fraglist: off [fixed]
tx-vlan-hw-insert: off [fixed]
rx-vlan-hw-parse: off [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-generic-segmentation: on
rx-gro: on
rx-lro: off [fixed]
tx-tcp-segmentation: on
tx-gso-robust: on [fixed]
tx-tcp-ecn-segmentation: on
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off
tx-gso-list: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
rx-ntuple-filter: off [fixed]
rx-hashing: off [fixed]
rx-checksum: on [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: on
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]
channels:
combined-max: 1
combined: 1
```
The available features depend on the network card and driver.
Some values are fixed by the driver and hardware and cannot be changed.
## Configuration
Use the [EthernetConfig](../../reference/configuration/network/ethernetconfig) document to configure Ethernet settings.
You can append the machine config document to the machine configuration (separating with `---`), or apply it as a [machine configuration patch](../../configure-your-talos-cluster/system-configuration/patching).
For example, to disable TCP segmentation on transmit:
```yaml theme={null}
apiVersion: v1alpha1
kind: EthernetConfig
name: enp0s2
features:
tx-tcp-segmentation: false
```
For rings and channels configuration, values can be increased if they do not exceed the maximum supported by the network card (the maximum values are reported in the status with `-max` suffix).
### Enable Wake-on-LAN support
Starting with 1.12, Talos Linux now supports configuring Wake-on-LAN (WOL) directly in the Ethernet configuration.
Use the `wakeOnLan` field under the `EthernetConfig` resource to enable WOL and specify the desired WOL modes.
This allows a node’s network interface to wake the system from a low-power state when receiving a magic packet or other WOL-triggering event.
Example:
```yaml theme={null}
apiVersion: v1alpha1
kind: EthernetConfig
name: enp0s3
wakeOnLan:
enabled: true
modes: ["magic", "unicast"]
```
#### Supported modes
Supported WOL modes depend on the NIC and driver, but common values include:
* `"magic"` — wake on magic packet (most common)
* `"unicast"` — wake on unicast packets
* `"multicast"` — wake on multicast packets
* `"broadcast"` — wake on broadcast packets
* `"phy"` — wake on PHY activity
> **Note**: If no modes are specified and `enabled: true`, Talos defaults to using the "magic" mode when supported by the hardware.
>
> WOL functionality requires that your hardware and BIOS/UEFI firmware support it and that it is not disabled there.
You can check WOL support and status with:
```bash theme={null}
talosctl get ethernetstatus -o yaml
```
This command will show the current wakeOnLan settings along with other Ethernet features.
# Routing Rules
Source: https://docs.siderolabs.com/talos/v1.13/networking/advanced/routing-rules
How to configure Linux routing rules for the machine.
Routing rules are used to control the routing of network traffic on the machine based on various criteria such as source and destination addresses, interfaces, and more.
Routing rules are evaluated in order of priority, and the first matching rule is applied to the traffic.
This allows for complex routing configurations, such as policy-based routing, where traffic can be routed differently based on its characteristics.
By default, a predefined routing table main is used for all traffic, but additional routing tables can be created and used with routing rules to achieve more granular control over traffic routing.
Routing rules can be configured in Talos Linux using the [`RoutingRuleConfig`](../../reference/configuration/network/routingruleconfig) machine configuration document:
```yaml theme={null}
apiVersion: v1alpha1
kind: RoutingRuleConfig
name: "1000"
src: 10.0.0.0/8
table: "100"
action: unicast
```
The `name` field specifies the priority of the routing rule, with lower numbers having higher priority.
The priority must be a number between 1 and 32764, where 1 is the highest priority and 32764 is the lowest priority.
Priorities 0, 32765, and 32766 are reserved for special use by the system and cannot be used for custom routing rules.
Once the routing rule is applied, any traffic with a source address in the `10.0.0.0/8` subnet will be routed according to the routing table `100` instead of the default `main` routing table, and the action `unicast` specifies that the traffic should be routed normally
(as opposed to being dropped or sent to a specific interface).
Routes in the routing table can be populated using static routes as shown in [static link configuration](./../configuration/static) by specifying the `table:` field (which defaults to `main`).
# Virtual (shared) IP
Source: https://docs.siderolabs.com/talos/v1.13/networking/advanced/vip
Using Talos Linux to set up a floating virtual IP address for cluster access.
One of the pain points when building a high-availability controlplane
is giving clients a single IP or URL at which they can reach any of the controlplane nodes.
The most common approaches - reverse proxy, load
balancer, BGP, and DNS - all require external resources, and add complexity in setting up Kubernetes.
To simplify cluster creation, Talos Linux supports a "Virtual" IP (VIP) address to access the Kubernetes API server, providing high availability with no other resources required.
What happens is that the controlplane machines vie for control of the shared IP address using etcd elections.
There can be only one owner of the IP address at any given time.
If that owner disappears or becomes non-responsive, another owner will be chosen,
and it will take up the IP address.
### Requirements
The controlplane nodes must share a layer 2 network, and the virtual IP must be assigned from that shared network subnet.
In practical terms, this means that they are all connected via a switch, with no router in between them.
Note that the virtual IP election depends on `etcd` being up, as Talos uses `etcd` for elections and leadership (control) of the IP address.
The virtual IP is not restricted by ports - you can access any port that the control plane nodes are listening on, on that IP address.
Thus it *is* possible to access the Talos API over the VIP, but it is *not recommended*, as you cannot access the VIP when etcd is down - and then you could not access the Talos API to recover etcd.
## Video walkthrough
To see a live demo of this writeup, see the video below:
## Choose your shared IP
The Virtual IP should be a reserved, unused IP address in the same subnet as
your controlplane nodes.
It should not be assigned or assignable by your DHCP server.
For our example, we will assume that the controlplane nodes have the following
IP addresses:
* `192.168.0.10`
* `192.168.0.11`
* `192.168.0.12`
We then choose our shared IP to be:
* `192.168.0.15`
## Configure your Talos machines
The shared IP setting is only valid for controlplane nodes.
For the example above, patch each of the controlplane nodes with the following machine configuration snippet ([Layer2VIPConfig](../../reference/configuration/network/layer2vipconfig)):
```yaml theme={null}
apiVersion: v1alpha1
kind: Layer2VIPConfig
name: 192.168.0.15
link: enp0s2
```
Virtual IP's can also be configured on a logical link (such as a VLAN); just update the configuration accordingly.
## Caveats
Since VIP functionality relies on `etcd` for elections, the shared IP will not come
alive until after you have bootstrapped Kubernetes.
Don't use the VIP as the `endpoint` in the `talosconfig`, as the VIP is bound to `etcd` and `kube-apiserver` health, and you will not be able to recover from a failure of either of those components using Talos API.
## VIP failover behavior
When the control plane node holding the VIP shuts down gracefully, the address is reassigned almost instantly, ensuring uninterrupted access.
However, if the node fails unexpectedly, for example, due to a power loss or crash, the failover process takes longer, typically up to a minute.
This slower response is by design as Talos coordinates VIP ownership through an etcd election process that must balance speed with safety.
The delay ensures that a temporary network hiccup or brief pause in communication does not lead to multiple nodes believing they own the VIP at the same time, a dangerous split-brain scenario.
By waiting out the election timeout before reassigning the VIP, Talos guarantees that only one node will advertise the shared IP, even if it means failover is slower in sudden failure scenarios.
### Impact on workloads
A VIP failover impacts only external access to the cluster, such as when you run `kubectl` against the API server.
Inside the cluster, workloads continue to function normally.
With the default configuration using [KubePrism and discovery](../../../../kubernetes-guides/advanced-guides/kubeprism), pods can still reach the Kubernetes API through service discovery, so they remain unaffected by the VIP status.
For external clients, however, a failover can briefly interrupt connectivity.
Long-lived connections, such as HTTP/2 sessions, may be broken when the VIP moves to a new node, and clients should be ready to reconnect once the failover is complete.
# VRF
Source: https://docs.siderolabs.com/talos/v1.13/networking/advanced/vrf
How to configure VRF (Virtual Routing and Forwarding) for your network.
VRF (Virtual Routing and Forwarding) is a [Linux feature](https://docs.kernel.org/networking/vrf.html) that allows you to create multiple separate routing tables on the same machine, effectively creating multiple virtual routers.
VRF device is associated with a specific routing table, and any network interface attached to the VRF device will use that routing table for its traffic.
VRFs can be configured in Talos Linux using the [`VRFConfig`](../../reference/configuration/network/vrfconfig) machine configuration document:
```yaml theme={null}
apiVersion: v1alpha1
kind: VRFConfig
name: vrf-blue
links:
- eno1
- eno5
table: "123"
up: true
addresses:
- address: 1.2.3.5/32
```
Addresses and routes can be configured in the same document as shown in [static link configuration](./../configuration/static).
The `links` field specifies the interfaces assigned to the VRF, and [link aliases](./../configuration/aliases) can be used here as well.
# Wireguard
Source: https://docs.siderolabs.com/talos/v1.13/networking/advanced/wireguard
Learn how to configure Wireguard link.
See this official Wireguard [quick start tutorial](https://www.wireguard.com/quickstart/) to understand the basic concepts.
For each machine, generate a public/private key pair:
```bash theme={null}
wg genkey | tee privatekey | wg pubkey > publickey
```
Wireguard network requires a set of overlay addresses that will be used by Wireguard interfaces on each machine ([WireguardConfig](../../reference/configuration/network/wireguardconfig)).
For example, you can use the `10.0.0.0/24` network for overlay addresses (if this network does not conflict with your existing networks).
```yaml theme={null}
apiVersion: v1alpha1
kind: WireguardConfig
name: wg.int
privateKey: GA1E1VB+g41Dl0+UH2TMW9C5953y+moVg6JIIqkJbmw= # private key of this machine
listenPort: 5042 # optional
peers:
- publicKey: 735jkJdcVDninU5PzLJ/S+bfN6Q3QOk6svWrVLMJQAk= # public key of another machine
allowedIPs:
- 10.0.0.2/32 # overlay address of another machine
endpoint: 192.168.1.2:5042 # endpoint to connect to, only one of the peers needs to specify this
- publicKey: uvdlJNva1X8/OCOZM+0gGT4Yu9x20odd3AWbbQUF7nM= # public key of yet another machine
allowedIPs:
- 10.0.0.3/32 # overlay address of yet another machine
up: true
addresses:
- address: 10.0.0.1/32 # overlay address of this machine
```
This configuration creates a Wireguard interface named `wg.int` with the overlay address `10.0.0.1/32`, if applied on all machines,
the machines will be able to communicate with each other over the Wireguard network using the overlay addresses.
If you want to route specific networks over the Wireguard interface, you need to set up routing accordingly.
See [KubeSpan](./../kubespan) for a way to make Talos Linux set up Wireguard overlay mesh network automatically across the cluster.
# Link Aliases
Source: https://docs.siderolabs.com/talos/v1.13/networking/configuration/aliases
Learn how to provide alternative names for network interfaces.
Talos Linux by default names physical network interfaces based on [predictable interface naming conventions](../predictable-interface-names) (e.g., `ens3`, `enp0s31f6`, etc.).
In some scenarios, it might be desirable to provide alternative, more user-friendly names for network interfaces.
This can be achieved using [LinkAliasConfig](../../reference/configuration/network/linkaliasconfig) configuration documents.
Note:
* only physical links can be aliased (logical links like bridges, bonds, or VLANs are given user-defined names directly);
* link alias configuration document should match exactly one physical link, otherwise the configuration will not be applied;
* link alias can be used interchangeably with physical link names in all networking configuration sections (links, routes, addresses, etc.).
The link alias is built using a CEL expression running on the specfication of a `LinkStatus` resource:
```yaml theme={null}
# talosctl get linkstatus enp0s2 -o yaml
...
spec:
altNames:
- enxb6dce1a4a634
index: 8
type: ether
linkIndex: 0
flags: UP,BROADCAST,RUNNING,MULTICAST,LOWER_UP
hardwareAddr: b6:dc:e1:a4:a6:34
permanentAddr: b6:dc:e1:a4:a6:34
broadcastAddr: ff:ff:ff:ff:ff:ff
mtu: 1500
queueDisc: pfifo_fast
operationalState: up
kind: ""
slaveKind: ""
busPath: "0000:00:02.0"
driver: virtio_net
driverVersion: 1.0.0
productID: "0x1000"
vendorID: "0x1af4"
product: Virtio network device
vendor: Red Hat, Inc.
linkState: true
speedMbit: 4294967295
port: Other
duplex: Unknown
```
The following example creates a link alias `net0` for the only physical network interface found on the system:
```yaml theme={null}
apiVersion: v1alpha1
kind: LinkAliasConfig
name: net0
selector:
match: true # as there is a single link only, we can match it this way
```
The following example creates a link alias `mgmt` for the network interface with a specific MAC address:
```yaml theme={null}
apiVersion: v1alpha1
kind: LinkAliasConfig
name: mgmt
selector:
match: mac(link.permanent_addr) == "00:1a:2b:3c:4d:5e"
```
The aliases can be observed in the `LinkStatus` resources:
```bash theme={null}
$ talosctl get links
NODE NAMESPACE TYPE ID VERSION ALIAS TYPE KIND HW ADDR OPER STATE LINK STATE
172.20.0.2 network LinkStatus enp0s2 4 net0 ether b6:dc:e1:a4:a6:34 up true
...
```
In the above example, the physical interface `enp0s2` has been aliased to `net0`, which can now be used in all networking configuration sections.
To see all aliases produced from the machine configuration, use the following:
```bash theme={null}
$ talosctl get linkaliasspec
NODE NAMESPACE TYPE ID VERSION ALIAS
172.20.0.2 network LinkAliasSpec enp0s2 1 net0
```
# Dynamic Addressing (DHCP)
Source: https://docs.siderolabs.com/talos/v1.13/networking/configuration/dynamic
Learn how to configure dynamic addresses and routes on the link using DHCP.
Talos Linux runs by default a DHCP client on all physical network interfaces to obtain IP addresses and other network settings automatically.
Once any explicit link configuration is applied, the default DHCP behavior is disabled, and DHCP must be explicitly enabled on the desired link(s).
DHCP allows to configure addresses, routes, DNS, and NTP settings dynamically from a DHCP server.
DHCP client can be enabled on physical and logical links (bridges, bonds, VLANs).
There are two DHCP versions supported in Talos Linux: DHCPv4 and DHCPv6.
## DHCPv4 configuration
To enable DHCPv4 on a physical link, create a [DHCPv4Config](../../reference/configuration/network/dhcpv4config) configuration document with the name of the link:
```yaml theme={null}
apiVersion: v1alpha1
kind: DHCPv4Config
name: enp0s3
```
Additional settings can be configured:
* `ignoreHostname: true`: ignore the hostname provided by the DHCP server.
* `routeMetric: 512`: set the metric for routes provided by DHCP.
* `clientIdentifier`: configures the DHCP client identifier, which can have one of the following values:
* `mac`: use the MAC address as the client identifier (default).
* `duid`: use a DUID (DHCP Unique Identifier) as the client identifier, requires `duidRaw` field to be set.
* `off`: disable the client identifier.
## DHCPv6 configuration
To enable DHCPv6 on a physical link, create a [DHCPv6Config](../../reference/configuration/network/dhcpv6config) configuration document with the name of the link:
```yaml theme={null}
apiVersion: v1alpha1
kind: DHCPv6Config
name: enp0s3
```
Additional settings are identical to DHCPv4.
## Observing status
Use `talosctl` to get the list of all configured operators (which includes DHCP clients):
```bash theme={null}
$ talosctl get operatorspecs
NODE NAMESPACE TYPE ID VERSION
172.20.0.2 network OperatorSpec dhcp4/enp0s2 1
```
To see operator configuration broken down by configuration source, add `--namespace=network-config` flag:
```bash theme={null}
$ talosctl get operatorspecs --namespace=network-config
NODE NAMESPACE TYPE ID VERSION
172.20.0.2 network-config OperatorSpec configuration/dhcp4/enp0s2 1
```
# Hostname
Source: https://docs.siderolabs.com/talos/v1.13/networking/configuration/hostname
Learn how to configure the hostname for your Talos nodes.
By default, the Talos Linux generates a default stable hostname for each machine based on machine ID like `talos-2ld-dvw`.
This default hostname can be overridden by DHCP-provided hostname, platform (cloud) metadata, or manually configured hostname.
## Configuration
The default configuration document for the hostname looks like this ([HostnameConfig](../../reference/configuration/network/hostnameconfig)):
```yaml theme={null}
apiVersion: v1alpha1
kind: HostnameConfig
auto: stable
```
The `auto` field can have the following values:
* `stable`: generates a stable hostname based on machine ID (default).
* `off`: disables automatic hostname generation, resulting in Talos waiting for another source (DHCP, cloud metadata, manual configuration) to set the hostname.
To manually set a specific hostname, create a `HostnameConfig` document with the desired hostname:
```yaml theme={null}
apiVersion: v1alpha1
kind: HostnameConfig
hostname: my-custom-hostname
```
When patching default machine config to set a custom hostname, use the following patch (auto hostname should be disabled):
```yaml theme={null}
apiVersion: v1alpha1
kind: HostnameConfig
hostname: my-custom-hostname
auto: off
```
## Observing status
Use `talosctl` to get the current hostname of a node:
```bash theme={null}
$ talosctl get hostname
NODE NAMESPACE TYPE ID VERSION HOSTNAME DOMAINNAME
172.20.0.2 network HostnameStatus hostname 3 talos-default-controlplane-1
```
To see all hostname configuration sources, use the following:
```bash theme={null}
$ talosctl get hostnamespec --namespace=network-config
NODE NAMESPACE TYPE ID VERSION
172.20.0.2 network-config HostnameSpec default/hostname 1
172.20.0.2 network-config HostnameSpec dhcp4/enp0s2/hostname 1
```
# Overview
Source: https://docs.siderolabs.com/talos/v1.13/networking/configuration/overview
Learn how to configure networking with Talos Linux.
Talos Linux is API-centric, so it is critical to establish network connectivity to be able to manage and operate Talos nodes.
Some network configuration is required to establish basic connectivity, while more advanced configuration can be applied to optimize networking for specific use-cases.
There are a few key concepts to understand when configuring networking with Talos Linux:
* **Interfaces (Links)**: Talos allows configuring multiple network interfaces per node. Each interface can be configured with static IPs, DHCP, or other methods.
* **Addresses**: IP addresses (both IPv4 and IPv6) can be assigned to interfaces. Talos supports both static and dynamic addressing.
* **Routes**: Custom routes can be defined to control traffic flow within the cluster and to external networks.
* **DNS**: Talos supports configuring DNS servers and search domains to ensure proper name resolution.
* **NTP**: Talos allows configuring NTP servers to ensure accurate time synchronization across nodes.
* **Advanced Networking**: Talos supports advanced networking features such as Wireguard for secure VPN connections, and SideroLink for out-of-band management.
* **Bridging and Bonding**: Talos supports network bridging and bonding for high availability and load balancing.
There are cases when there is no need to configure networking at all:
* By default, if no network configuration is provided, Talos will attempt to use DHCP on the primary interface to obtain an IP address and other network settings.
This is often sufficient for basic connectivity in many environments.
* In cloud environments, Talos can leverage cloud-init or similar mechanisms to automatically configure networking based on the cloud provider's metadata service.
The hardest network configuration scenario involves setting up a Talos machine in an isolated environment without DHCP or any external network services which requires
fully manual configuration of interfaces, addresses, routes, DNS, and NTP.
In this case, a configuration source which doesn't rely on network connectivity is required, such as an embedded configuration, or cloud-init user-data bundled with the Talos image.
Talos Linux supports network configuration coming from multiple sources, and it is recommended to use the machine configuration as the primary source of truth for networking settings.
But other configuration sources can be used to override or supplement the machine configuration as needed: kernel arguments, cloud-specific data sources, dashboard metal network settings, etc.
> Note: Default Talos Linux behavior to run DHCP on any physical link is disabled automatically when any link is configured explicitly in the machine configuration.
> In order to keep DHCP running, it must be explicitly enabled on the desired link(s).
# Physical Links
Source: https://docs.siderolabs.com/talos/v1.13/networking/configuration/physical
Learn how to configure physical network interfaces.
## Configuration
Physical network interfaces (links) can be configured in Talos Linux using the [LinkConfig](../../reference/configuration/network/linkconfig) configuration objects.
The following properties can be configured for physical links:
* MTU (`mtu`): sets the Maximum Transmission Unit for the interface (default is 1500).
* bring the link up or down (`up`): controls whether the interface is administratively up or down.
Here is an example of configuring a physical network interface with a specific MTU and bringing it up:
```yaml theme={null}
apiVersion: v1alpha1
kind: LinkConfig
name: enp0s3
mtu: 9000
up: true
```
[Link aliases](./aliases) can also be used to refer to physical links in a more user-friendly way:
```yaml theme={null}
apiVersion: v1alpha1
kind: LinkAliasConfig
name: mgmt
selector:
match: mac(link.permanent_addr) == "00:1a:2b:3c:4d:5e"
---
apiVersion: v1alpha1
kind: LinkConfig
name: mgmt
mtu: 9000
up: true
```
For low-level control over physical link properties, such as offloading features, refer to the [Ethernet configuration](./../advanced/ethernet-config) documentation.
## Observing status
Use `talosctl` to observe the status of all links:
```bash theme={null}
$ talosctl get links
NODE NAMESPACE TYPE ID VERSION ALIAS TYPE KIND HW ADDR OPER STATE LINK STATE
172.20.0.2 network LinkStatus bond0 1 ether bond 3e:f0:5c:84:4b:5e down false
172.20.0.2 network LinkStatus dummy0 1 ether dummy 1e:e5:74:54:52:21 down false
172.20.0.2 network LinkStatus enp0s2 3 net0 ether ae:48:3d:ca:38:28 up true
172.20.0.2 network LinkStatus flannel.1 2 ether vxlan fa:1a:43:6d:23:e1 unknown true
172.20.0.2 network LinkStatus ip6tnl0 1 tunnel6 ip6tnl 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 down false
172.20.0.2 network LinkStatus lo 2 loopback 00:00:00:00:00:00 unknown true
172.20.0.2 network LinkStatus sit0 1 sit sit 00:00:00:00 down false
172.20.0.2 network LinkStatus teql0 1 void down false
172.20.0.2 network LinkStatus tunl0 1 ipip ipip 00:00:00:00 down false
```
The output shows all network links, their operational state, and other relevant information.
To see the list of links managed by Talos, use the following:
```bash theme={null}
$ talosctl get linkspecs
NODE NAMESPACE TYPE ID VERSION
172.20.0.2 network LinkSpec enp0s2 6
172.20.0.2 network LinkSpec lo 2
```
To see the list of link configuration broken down by configuration source, add `--namespace=network-config` flag:
```bash theme={null}
$ talosctl get links --namespace=networkconfig
NODE NAMESPACE TYPE ID VERSION
172.20.0.2 network-config LinkSpec configuration/enp0s2 1
172.20.0.2 network-config LinkSpec default/lo 1
172.20.0.2 network-config LinkSpec dhcp4/enp0s2/enp0s2 1
```
# Resolvers
Source: https://docs.siderolabs.com/talos/v1.13/networking/configuration/resolvers
Learn how to configure DNS resolvers.
DNS resolvers are used by Talos to resolve domain names into IP addresses.
By default, Talos will use DNS resolvers `8.8.8.8` and `1.1.1.1`, and DNS servers might be provided via DHCP or cloud platform metadata.
## Configuration
To configure custom DNS resolvers, create a [ResolverConfig](../../reference/configuration/network/resolverconfig) document like this:
```yaml theme={null}
apiVersion: v1alpha1
kind: ResolverConfig
nameservers:
- address: 10.0.0.1
- address: 2001:4860:4860::8888
searchDomains: # optional
domains:
- example.org
- example.com
disableDefault: false
```
The `nameservers` field is a list of DNS server IP addresses that Talos will use for DNS resolution.
The `searchDomains` field allows you to specify search domains that will be appended to unqualified domain names during DNS resolution, the default search domains is to use the domain part of the machine's hostname.
The `disableDefault` field, when set to `true`, prevents Talos from using the default search domains derived from the machine's hostname.
See [Host DNS](../host-dns) for more information about DNS resolution in Talos.
## Observing status
Use `talosctl` to get the current resolver configuration of a node:
```bash theme={null}
$ talosctl get resolvers
NODE NAMESPACE TYPE ID VERSION RESOLVERS SEARCH DOMAINS
172.20.0.2 network ResolverStatus resolvers 4 ["172.20.0.1"] []
```
To see all resolver configuration sources, use the following:
```bash theme={null}
$ talosctl get resolverspec --namespace=network-config
NODE NAMESPACE TYPE ID VERSION LAYER RESOLVERS SEARCH DOMAINS
172.20.0.2 network-config ResolverSpec default/resolvers 1 default ["1.1.1.1","8.8.8.8"]
172.20.0.2 network-config ResolverSpec dhcp4/enp0s2/resolvers 1 operator ["172.20.0.1"]
```
# Static Addressing
Source: https://docs.siderolabs.com/talos/v1.13/networking/configuration/static
Learn how to configure static addresses and routes on the link.
Static addressing allows you to assign fixed IP addresses to network interfaces in Talos Linux.
Static configuration is available for physical and logical links (bridges, bonds, VLANs).
For physical links, configuration is attached to the `LinkConfig` document, while for logical links, configuration is attached directly to the logical link document.
## Configuration
Addresses can be specified in the `addresses` field as a list of CIDR (Classless Inter-Domain Routing) notation strings:
```yaml theme={null}
addresses:
- address: 192.168.1.100/24
- address: 2001:db8::1/64
routePriority: 100
```
The `routePriority` field is optional and can be used to set the priority of the implicit route associated with the address.
Routes can be specified in the `routes` field as a list of route objects:
```yaml theme={null}
routes:
- destination: 10.3.5.0/24
gateway: 10.3.5.1
- gateway: fe80::1
```
Each route object can include the following fields:
* `destination`: The destination network in CIDR notation. If omitted, a default route is created.
* `gateway`: The gateway IP address for the route. If omitted, the route is considered directly connected.
* additional advanced fields like `metric`, `mtu` can be specified as needed.
There are some typical configuration scenarios to consider:
* address with a subnet and a default route via the gateway from the same subnet:
```yaml theme={null}
addresses:
- address: 192.168.1.100/24
routes:
- gateway: 192.168.1.1
```
With this configuration, all addresses 192.168.1.1-192.168.1.254 are reachable directly, while all other addresses are reachable via the gateway
192.168.1.1.
* a machine is given an address from a subnet, but the default gateway is outside of that subnet:
```yaml theme={null}
addresses:
- address: 192.168.1.100/32
routes:
- destination: 192.168.1.1
- gateway: 192.168.1.1
```
In this example, the machine has a /32 address, so no other addresses are reachable directly.
A specific route to the gateway 192.168.1.1 is required to reach other networks.
* a private network which doesn't have a default gateway, but needs specific routes to other networks:
```yaml theme={null}
addresses:
- address: 10.0.0.3/24
routes:
- destination: 10.1.0.0/24
gateway: 10.0.0.1
```
A complete example of static addressing configuration for a physical link:
```yaml theme={null}
apiVersion: v1alpha1
kind: LinkConfig
name: enp0s3
addresses:
- address: 192.168.1.100/24
routes:
- gateway: 192.168.1.1
```
## Observing status
You can observe the status of addresses and routes using `talosctl`:
```bash theme={null}
$ talosctl get addresses
NODE NAMESPACE TYPE ID VERSION ADDRESS LINK
172.20.0.2 network AddressStatus enp0s2/172.20.0.2/24 1 172.20.0.2/24 enp0s2
172.20.0.2 network AddressStatus enp0s2/fe80::ac48:3dff:feca:3828/64 2 fe80::ac48:3dff:feca:3828/64 enp0s2
172.20.0.2 network AddressStatus lo/127.0.0.1/8 1 127.0.0.1/8 lo
172.20.0.2 network AddressStatus lo/169.254.116.108/32 1 169.254.116.108/32 lo
172.20.0.2 network AddressStatus lo/::1/128 1 ::1/128 lo
```
And same for routes:
```bash theme={null}
NODE NAMESPACE TYPE ID VERSION DESTINATION GATEWAY LINK METRIC
172.20.0.2 network RouteStatus enp0s2/inet6//fe80::/64/256 1 fe80::/64 enp0s2 256
172.20.0.2 network RouteStatus inet4//172.20.0.0/24/1024 1 172.20.0.0/24 enp0s2 1024
172.20.0.2 network RouteStatus inet4/172.20.0.1//1024 1 172.20.0.1 enp0s2 1024
172.20.0.2 network RouteStatus local/enp0s2/inet6//fe80::/128/0 1 fe80::/128 enp0s2 0
172.20.0.2 network RouteStatus local/enp0s2/inet6//fe80::ac48:3dff:feca:3828/128/0 1 fe80::ac48:3dff:feca:3828/128 enp0s2 0
172.20.0.2 network RouteStatus local/enp0s2/inet6//ff00::/8/256 1 ff00::/8 enp0s2 256
172.20.0.2 network RouteStatus local/inet4//127.0.0.0/8/0 1 127.0.0.0/8 lo 0
172.20.0.2 network RouteStatus local/inet4//127.0.0.1/32/0 1 127.0.0.1/32 lo 0
172.20.0.2 network RouteStatus local/inet4//127.255.255.255/32/0 1 127.255.255.255/32 lo 0
172.20.0.2 network RouteStatus local/inet4//169.254.116.108/32/0 1 169.254.116.108/32 lo 0
172.20.0.2 network RouteStatus local/inet4//172.20.0.2/32/0 1 172.20.0.2/32 enp0s2 0
172.20.0.2 network RouteStatus local/inet4//172.20.0.255/32/0 1 172.20.0.255/32 enp0s2 0
172.20.0.2 network RouteStatus local/lo/inet6//::1/128/0 1 ::1/128 lo 0
```
This shows all configured addresses and routes, including those automatically created for link-local addresses, managed by CNI plugins, or other system components.
To see the list of addresses and routes managed by Talos, use the following:
```bash theme={null}
$ talosctl get addressspecs
NODE NAMESPACE TYPE ID VERSION
172.20.0.2 network AddressSpec enp0s2/172.20.0.2/24 2
172.20.0.2 network AddressSpec lo/127.0.0.1/8 2
172.20.0.2 network AddressSpec lo/169.254.116.108/32 2
$ talosctl get routespecs
NODE NAMESPACE TYPE ID VERSION
172.20.0.2 network RouteSpec inet4/172.20.0.1//1024 2
```
In order to see managed configuration with information about various sources (static config, DHCP, CNI, etc.) add `--namespace=networkconfig` flag:
```bash theme={null}
$ talosctl get addressspecs --namespace network-config
NODE NAMESPACE TYPE ID VERSION
172.20.0.2 network-config AddressSpec default/lo/127.0.0.1/8 1
172.20.0.2 network-config AddressSpec dhcp4/enp0s2/enp0s2/172.20.0.2/24 1
172.20.0.2 network-config AddressSpec operator/lo/169.254.116.108/32 1
$ talosctl get routespecs --namespace network-config
NODE NAMESPACE TYPE ID VERSION
172.20.0.2 network-config RouteSpec dhcp4/enp0s2/inet4/172.20.0.1//1024 1
```
# Time Servers
Source: https://docs.siderolabs.com/talos/v1.13/networking/configuration/time
Learn how to configure time servers (NTP sync).
Talos Linux defaults to using `time.cloudflare.com` as the NTP server for time synchronization.
## Configuration
To customize the NTP servers used by Talos, create a [TimeSyncConfig](../../reference/configuration/network/timesyncconfig) document like:
```yaml theme={null}
apiVersion: v1alpha1
kind: TimeSyncConfig
ntp:
servers:
- 0.pool.ntp.org
- 1.pool.ntp.org
```
See [Time Sync](../../configure-your-talos-cluster/system-configuration/time-sync) for more details about time synchronization in Talos Linux.
## Observing status
Use `talosctl` to get the current time synchronization configuration of a node:
```bash theme={null}
$ talosctl get timeservers
NODE NAMESPACE TYPE ID VERSION TIMESERVERS
172.20.0.2 network TimeServerStatus timeservers 1 ["time.cloudflare.com"]
```
To see all time server configuration sources, use the following:
```bash theme={null}
$ talosctl get timeserverspec --namespace=network-config
NODE NAMESPACE TYPE ID VERSION
172.20.0.2 network-config TimeServerSpec default/timeservers 1
```
# Corporate Proxies
Source: https://docs.siderolabs.com/talos/v1.13/networking/corporate-proxies
How to configure Talos Linux to use proxies in a corporate environment
## Appending the certificate authority of MITM proxies
See [Custom Certificate Authorities](../security/certificate-authorities) to append the CA certificate of your corporate proxy to the trusted store.
## Configuring a machine to use the proxy
To configure Talos to use an HTTP or HTTPS proxy, set the appropriate environment variables in the machine configuration:
```yaml theme={null}
machine:
env:
http_proxy:
https_proxy:
no_proxy:
```
In proxy-restricted environments, you may also need to configure DNS (`nameservers`) and NTP (`timeservers`) explicitly:
```yaml theme={null}
machine:
env:
...
time:
servers:
-
-
-
...
network:
nameservers:
-
-
-
```
If proxy access is required before the machine configuration is applied (for example, during initial boot), provide the proxy settings via [kernel command line arguments](../reference/kernel):
```text theme={null}
talos.environment=http_proxy= talos.environment=https_proxy=
```
DNS and NTP settings can also be managed using their dedicated configuration documents: [ResolverConfig](../reference/configuration/network/resolverconfig) and [TimeSyncConfig](../reference/configuration/network/timesyncconfig) respectively.
# Egress Domains
Source: https://docs.siderolabs.com/talos/v1.13/networking/egress-domains
Allowing outbound access for installing Talos
For some more constrained environments, it is important to whitelist only specific domains for outbound internet access.
These rules will need to be updated to allow for certain domains if the user wishes to still install and bootstrap Talos from public sources.
That said, users should also note that all of the following components can be mirrored locally with an internal registry, as well as a self-hosted [discovery service](https://github.com/siderolabs/discovery-service) and [image factory](https://github.com/siderolabs/image-factory).
Discovery Service:
* `discovery.talos.dev`
Image Factory:
* `factory.talos.dev`
* `*.factory.talos.dev`
* `*.r2.cloudflarestorage.com` (Cloudflare R2 origin storage)
Google Container Registry / Google Artifact Registry (GCR/GAR):
* `gcr.io`
* `storage.googleapis.com` (backing blob storage for images)
* `*.pkg.dev` (backing blob storage for images)
Github Container Registry (GHCR)
* `ghcr.io`
* `*.githubusercontent.com` (backing blob storage for images)
Kubernetes Registry (k8s.io)
* `registry.k8s.io`
* `*.s3.dualstack.us-east-1.amazonaws.com` (backing blob storage for images)
These rules only cover that which is required for Talos to be fully installed and running. There may be other domains like `docker.io` that must be allowed for non-default CNIs or workload container images.
# Host DNS
Source: https://docs.siderolabs.com/talos/v1.13/networking/host-dns
How to configure Talos host DNS caching server.
Talos Linux starting with 1.7.0 provides a caching DNS resolver for host workloads (including host networking pods).
Host DNS resolver is enabled by default for clusters created with Talos 1.7, and it can be enabled manually on upgrade.
## Enabling host DNS
Use the following machine configuration patch to enable host DNS resolver:
```yaml theme={null}
machine:
features:
hostDNS:
enabled: true
```
Host DNS can be disabled by setting `enabled: false` as well.
> Note: When disabling host DNS, `forwardKubeDNSToHost` must also be set to `false`.
> Setting `forwardKubeDNSToHost: true` while `hostDNS.enabled` is `false` will result in a configuration validation error.
## Operations
When enabled, Talos Linux starts a DNS caching server on the host, listening on address `127.0.0.53:53` (both TCP and UDP protocols).
The host `/etc/resolv.conf` file is rewritten to point to the host DNS server:
```shell theme={null}
$ talosctl read /etc/resolv.conf
nameserver 127.0.0.53
```
All host-based workloads will use the host DNS server for name resolution.
Host DNS server forwards requests to the upstream DNS servers, which are either acquired automatically (DHCP, platform sources, kernel args), or specified in the machine configuration.
The upstream DNS servers can be observed with:
```shell theme={null}
$ talosctl get resolvers
NODE NAMESPACE TYPE ID VERSION RESOLVERS
172.20.0.2 network ResolverStatus resolvers 2 ["8.8.8.8","1.1.1.1"]
```
Logs of the host DNS resolver can be queried with:
```shell theme={null}
talosctl logs dns-resolve-cache
```
Upstream server status can be observed with:
```shell theme={null}
$ talosctl get dnsupstream
NODE NAMESPACE TYPE ID VERSION HEALTHY ADDRESS
172.20.0.2 network DNSUpstream 1.1.1.1 1 true 1.1.1.1:53
172.20.0.2 network DNSUpstream 8.8.8.8 1 true 8.8.8.8:53
```
## Forwarding `kube-dns` to host DNS
> Note: This feature is enabled by default for new clusters created with Talos 1.8.0 and later.
When host DNS is enabled, by default, `kube-dns` service (`CoreDNS` in Kubernetes) uses host DNS server to resolve external names.
This way the cache is shared between the host DNS and `kube-dns`.
Talos allows forwarding `kube-dns` to the host DNS resolver to be disabled with:
```yaml theme={null}
machine:
features:
hostDNS:
enabled: true
forwardKubeDNSToHost: false
```
This configuration should be applied to all nodes in the cluster, if applied after cluster creation, restart `coredns` pods in Kubernetes to pick up changes.
When `forwardKubeDNSToHost` is enabled, Talos Linux allocates IP address `169.254.116.108` for the host DNS server, and `kube-dns` service is configured to use this IP address as the upstream DNS server:
This way `kube-dns` service forwards all DNS requests to the host DNS server, and the cache is shared between the host and `kube-dns`.
## Resolving Talos cluster member names
Host DNS can be configured to resolve Talos cluster member names to IP addresses, so that the host can communicate with the cluster members by name.
Sometimes machine hostnames are already resolvable by the upstream DNS, but this might not always be the case.
Enabling the feature:
```yaml theme={null}
machine:
features:
hostDNS:
enabled: true
resolveMemberNames: true
```
When enabled, Talos Linux uses [discovery](../configure-your-talos-cluster/system-configuration/discovery) data to resolve Talos cluster member names to IP addresses:
With the example output above, `talos-default-worker-1` name will resolve to `172.20.0.3`.
Example usage:
```shell theme={null}
talosctl -n talos-default-worker-1 version
```
When combined with `forwardKubeDNSToHost`, `kube-dns` service will also resolve Talos cluster member names to IP addresses.
# Ingress Firewall
Source: https://docs.siderolabs.com/talos/v1.13/networking/ingress-firewall
Learn to use Talos Linux Ingress Firewall to limit access to the host services.
Talos Linux Ingress Firewall is a simple and effective way to limit network access to the services running on the host, which includes both Talos standard
services (e.g. `apid` and `kubelet`), and any additional workloads that may be running on the host.
Talos Linux Ingress Firewall doesn't affect the traffic between the Kubernetes pods/services, please use CNI Network Policies for that.
> Note: If you use another tool that provides node level network filtering (e.g. [Cilium Host Firewall](https://cilium.io/use-cases/host-firewall/)) it may take precedence in the nftables chain and bypass OS level rules.
## Configuration
Ingress rules are configured as extra documents [NetworkDefaultActionConfig](../reference/configuration/network/networkdefaultactionconfig) and
[NetworkRuleConfig](../reference/configuration/network/networkruleconfig) in the Talos machine configuration:
```yaml theme={null}
apiVersion: v1alpha1
kind: NetworkDefaultActionConfig
ingress: block
---
apiVersion: v1alpha1
kind: NetworkRuleConfig
name: kubelet-ingress
portSelector:
ports:
- 10250
protocol: tcp
ingress:
- subnet: 172.20.0.0/24
except: 172.20.0.1/32
```
The first document configures the default action for ingress traffic, which can be either `accept` or `block`, with the default being `accept`.
If the default action is set to `accept`, then all ingress traffic will be allowed, unless there is a matching rule that blocks it.
If the default action is set to `block`, then all ingress traffic will be blocked, unless there is a matching rule that allows it.
With either `accept` or `block`, traffic is always allowed on the following network interfaces:
* `lo`
* `siderolink`
* `kubespan`
In `block` mode:
* ICMP and ICMPv6 traffic is also allowed with a rate limit of 5 packets per second
* ICMP packet types timestamp-request, timestamp-reply, address-mask-request and address-mask-reply are blocked (CVE-1999-0524 mitigation)
* traffic between Kubernetes pod/service subnets is allowed (for native routing CNIs)
The second document defines an ingress rule for a set of ports and protocols on the host.
The `NetworkRuleConfig` might be repeated many times to define multiple rules, but each document must have a unique name.
The `ports` field accepts either a single port or a port range:
```yaml theme={null}
portSelector:
ports:
- 10250
- 10260
- 10300-10400
```
The `protocol` might be either `tcp` or `udp`.
The `ingress` specifies the list of subnets that are allowed to access the host services, with the optional `except` field to exclude a set of addresses from the subnet.
> Note: incorrect configuration of the ingress firewall might result in the host becoming inaccessible over Talos API.
> It is recommended that the configuration be [applied](../configure-your-talos-cluster/system-configuration/editing-machine-configuration) in `--mode=try` to ensure it is reverted in case of a mistake.
## Recommended rules
The following rules improve the security of the cluster and cover only standard Talos services.
If there are additional services running with host networking in the cluster, they should be covered by additional rules.
In `block` mode, the ingress firewall will also block encapsulated traffic (e.g. VXLAN) between the nodes, which needs to be explicitly allowed for the Kubernetes
networking to function properly.
Please refer to the documentation of the CNI in use for the specific ports required.
Some default configurations are listed below:
* Flannel, Calico: `vxlan` UDP port 4789
* Cilium: `vxlan` UDP port 8472
In the examples we assume the following template variables to describe the cluster:
* `$CLUSTER_SUBNET`, e.g. `172.20.0.0/24` - the subnet which covers all machines in the cluster
* `$CP1`, `$CP2`, `$CP3` - the IP addresses of the controlplane nodes
* `$VXLAN_PORT` - the UDP port used by the CNI for encapsulated traffic
### Control plane
In this example Ingress policy:
* `apid` and Kubernetes API are wide open
* `kubelet` and `trustd` API are only accessible within the cluster
* `etcd` API is limited to controlplane nodes
```yaml theme={null}
apiVersion: v1alpha1
kind: NetworkDefaultActionConfig
ingress: block
---
apiVersion: v1alpha1
kind: NetworkRuleConfig
name: kubelet-ingress
portSelector:
ports:
- 10250
protocol: tcp
ingress:
- subnet: $CLUSTER_SUBNET
---
apiVersion: v1alpha1
kind: NetworkRuleConfig
name: apid-ingress
portSelector:
ports:
- 50000
protocol: tcp
ingress:
- subnet: 0.0.0.0/0
- subnet: ::/0
---
apiVersion: v1alpha1
kind: NetworkRuleConfig
name: trustd-ingress
portSelector:
ports:
- 50001
protocol: tcp
ingress:
- subnet: $CLUSTER_SUBNET
---
apiVersion: v1alpha1
kind: NetworkRuleConfig
name: kubernetes-api-ingress
portSelector:
ports:
- 6443
protocol: tcp
ingress:
- subnet: 0.0.0.0/0
- subnet: ::/0
---
apiVersion: v1alpha1
kind: NetworkRuleConfig
name: etcd-ingress
portSelector:
ports:
- 2379-2380
protocol: tcp
ingress:
- subnet: $CP1/32
- subnet: $CP2/32
- subnet: $CP3/32
---
apiVersion: v1alpha1
kind: NetworkRuleConfig
name: cni-vxlan
portSelector:
ports:
- $VXLAN_PORT
protocol: udp
ingress:
- subnet: $CLUSTER_SUBNET
```
### Worker
* `kubelet` and `apid` API are only accessible within the cluster
```yaml theme={null}
apiVersion: v1alpha1
kind: NetworkDefaultActionConfig
ingress: block
---
apiVersion: v1alpha1
kind: NetworkRuleConfig
name: kubelet-ingress
portSelector:
ports:
- 10250
protocol: tcp
ingress:
- subnet: $CLUSTER_SUBNET
---
apiVersion: v1alpha1
kind: NetworkRuleConfig
name: apid-ingress
portSelector:
ports:
- 50000
protocol: tcp
ingress:
- subnet: $CLUSTER_SUBNET
---
apiVersion: v1alpha1
kind: NetworkRuleConfig
name: cni-vxlan
portSelector:
ports:
- $VXLAN_PORT
protocol: udp
ingress:
- subnet: $CLUSTER_SUBNET
```
## Learn more
Talos Linux Ingress Firewall uses `nftables` to perform the filtering.
With the default action set to `accept`, the following rules are applied (example):
```text theme={null}
table inet talos {
chain ingress {
type filter hook input priority filter; policy accept;
iifname { "lo", "siderolink", "kubespan" } accept
ip saddr != { 172.20.0.0/24 } tcp dport { 10250 } drop
meta nfproto ipv6 tcp dport { 10250 } drop
}
}
```
With the default action set to `block`, the following rules are applied (example):
```text theme={null}
table inet talos {
chain ingress {
type filter hook input priority filter; policy drop;
iifname { "lo", "siderolink", "kubespan" } accept
ct state { established, related } accept
ct state invalid drop
meta l4proto icmp type { timestamp-request, timestamp-reply, address-mask-request, address-mask-reply } drop
meta l4proto icmp limit rate 5/second accept
meta l4proto ipv6-icmp limit rate 5/second accept
ip saddr { 172.20.0.0/24 } tcp dport { 10250 } accept
meta nfproto ipv4 tcp dport { 50000 } accept
meta nfproto ipv6 tcp dport { 50000 } accept
}
}
```
The running `nftable` configuration can be inspected with `talosctl get nftableschain -o yaml`.
The Ingress Firewall documents can be extracted from the machine config with the following command:
`talosctl get mc v1alpha1 -o yaml | yq .spec | yq 'select(.kind == "NetworkDefaultActionConfig"),select(.kind == "NetworkRuleConfig" )'`
# KubeSpan
Source: https://docs.siderolabs.com/talos/v1.13/networking/kubespan
Learn to use KubeSpan to connect Talos Linux machines securely across networks.
KubeSpan is a feature of Talos that automates the setup and maintenance of a full mesh [WireGuard](https://www.wireguard.com) network for your cluster, giving you the ability to operate hybrid Kubernetes clusters that can span the edge, datacenter, and cloud.
Management of keys and discovery of peers can be completely automated, making it simple and easy to create hybrid clusters.
KubeSpan consists of client code in Talos Linux, as well as a [discovery service](../configure-your-talos-cluster/system-configuration/discovery) that enables clients to securely find each other.
Sidero Labs operates a free Discovery Service, but the discovery service may, with a commercial license, be operated by your organization and can be [downloaded here](https://github.com/siderolabs/discovery-service).
## Video walkthrough
To see a live demo of KubeSpan, see one the videos below:
## Network requirements
KubeSpan uses **UDP port 51820** to carry all KubeSpan encrypted traffic.
Because UDP traversal of firewalls is often lenient, and the Discovery Service communicates the apparent IP address of all peers to all other peers, KubeSpan will often work automatically, even when each node is behind its own firewall.
However, when both ends of a KubeSpan connection are behind firewalls, it is possible the connection may not be established correctly - it depends on each end sending out packets in a limited time window.
Thus best practice is to ensure that one end of all possible node-node communication allows UDP port 51820, inbound.
For example, if control plane nodes are running in a corporate data center, behind firewalls, KubeSpan connectivity will work correctly so long as worker nodes on the public Internet can receive packets on UDP port 51820.
(Note the workers will also need to receive TCP port 50000 for initial configuration via `talosctl`).
An alternative topology would be to run control plane nodes in a public cloud, and allow inbound UDP port 51820 to the control plane nodes.
Workers could be behind firewalls, and KubeSpan connectivity will be established.
Note that if workers are in different locations, behind different firewalls, the KubeSpan connectivity between workers *should* be correctly established, but may require opening the KubeSpan UDP port on the local firewall also.
## Caveats
### Kubernetes API endpoint limitations
When the K8s endpoint is an IP address that is **not** part of Kubespan, but is an address that is forwarded on to the Kubespan address of a control plane node, without changing the source address, then worker nodes will fail to join the cluster.
In such a case, the control plane node has no way to determine whether the packet arrived on the private Kubespan address, or the public IP address.
If the source of the packet was a Kubespan member, the reply will be Kubespan encapsulated, and thus not translated to the public IP, and so the control plane will reply to the session with the wrong address.
This situation is seen, for example, when the Kubernetes API endpoint is the public IP of a VM in GCP or Azure for a single node control plane.
The control plane will receive packets on the public IP, but will reply from it's KubeSpan address.
The workaround is to create a load balancer to terminate the Kubernetes API endpoint.
### Digital ocean limitations
Digital Ocean assigns an "Anchor IP" address to each droplet.
Talos Linux correctly identifies this as a link-local address, and configures KubeSpan correctly, but this address will often be selected by Flannel or other CNIs as a node's private IP.
Because this address is not routable, nor advertised via KubeSpan, it will break pod-pod communication between nodes.
This can be worked-around by assigning a non-Anchor private IP:
`kubectl annotate node do-worker flannel.alpha.coreos.com/public-ip-overwrite=10.116.X.X`
Then restarting flannel:
`kubectl delete pods -n kube-system -l k8s-app=flannel`
### Host port limitations
As mentioned in Network Requirements, Kubespan uses **UDP port 51820** to carry all KubeSpan encrypted traffic.
For clusters that make heavy use of host ports for Kubernetes pods, care should be taken to ensure that this port is not given to these pods.
Failure to do so can result in a pod being assigned the 51820 port and conflicting with Kubespan traffic.
### Cilium compatibility limitations
KubeSpan and Cilium can generally be used together, but some Cilium configurations require specific settings to avoid asymmetric routing.
As of Talos 1.13, Cilium's native routing mode works with KubeSpan when `advertiseKubernetesNetworks: true` is set.
Configure Cilium with the following BPF settings:
```yaml theme={null}
bpf:
masquerade: true # can be set to false if masquerading isn't needed
hostLegacyRouting: true # required — eBPF host routing bypasses the kernel routing table and conflicts with KubeSpan
```
If you don't need native routing, Cilium's default configuration works with KubeSpan out of the box.
For advanced eBPF features without these constraints, use Cilium's built-in WireGuard encryption instead of KubeSpan.
## Enabling
### Creating a new cluster
To enable KubeSpan for a new cluster, we can use the `--with-kubespan` flag in `talosctl gen config`.
This will enable peer discovery and KubeSpan.
```yaml theme={null}
cluster:
discovery:
enabled: true
# Configure registries used for cluster member discovery.
registries:
kubernetes: # Kubernetes registry is problematic with KubeSpan, if the control plane endpoint is routable itself via KubeSpan.
disabled: true
service: {}
---
apiVersion: v1alpha1
kind: KubeSpanConfig
enabled: true # Enable the KubeSpan feature.
```
> The default discovery service is an external service hosted by Sidero Labs at `https://discovery.talos.dev/`.
> Contact Sidero Labs if you need to run this service privately.
### Enabling for an existing cluster
In order to enable KubeSpan on an existing cluster, add `KubeSpanConfig` and `discovery` settings in the machine config for each machine in the cluster (`discovery` is enabled by default):
```yaml theme={null}
cluster:
discovery:
enabled: true
---
apiVersion: v1alpha1
kind: KubeSpanConfig
enabled: true
```
## Configuration
KubeSpan automatically discovers cluster members, exchanges WireGuard public keys, and establishes a full mesh network between nodes.
In most cases, no additional configuration is required. However, the following options are available for advanced use cases:
```yaml theme={null}
apiVersion: v1alpha1
kind: KubeSpanConfig
enabled: true
advertiseKubernetesNetworks: false
allowDownPeerBypass: false
mtu: 1420
filters:
endpoints:
- 0.0.0.0/0
- ::/0
excludeAdvertisedNetworks:
- 2007::/64
```
`advertiseKubernetesNetworks`
Controls whether the node advertises Kubernetes service and pod networks to other nodes over KubeSpan.
By default, this setting is disabled. When disabled, KubeSpan handles only node-to-node traffic, while pod-to-pod traffic is routed and encapsulated by the CNI plugin.
`allowDownPeerBypass`
Controls whether a node can bypass WireGuard when the destination is not connected through KubeSpan.
If this option is enabled, traffic may be routed without encryption when the destination is outside the KubeSpan network. However, it provides a workaround for situations where a node is not connected to KubeSpan but still needs access to the cluster.
`mtu`
Configures the WireGuard MTU. The default value is 1420.
This value is safe when the underlying network MTU is 1500.
If the underlying MTU is smaller, adjust accordingly:
```bash theme={null}
KubeSpan MTU = Underlying MTU - 80
```
### Filters
The `filters` setting provides fine-grained control over what this node advertises over KubeSpan.
Filters apply only to the local node.
`filters.endpoints`
Controls which node addresses are advertised as WireGuard endpoints.
By default, all detected addresses are advertised, and KubeSpan cycles through them until it finds one that works.
This is useful when:
* Some node IPs are unreachable from peers
* Nodes have multiple network paths and one should be preferred
**Example:**
```yaml theme={null}
apiVersion: v1alpha1
kind: KubeSpanConfig
filters:
endpoints:
- 0.0.0.0/0
- ::/0
- "!123.0.0.0/24"
```
`filters.excludeAdvertisedNetworks`
Excludes specific networks from being advertised over KubeSpan.
By default, all eligible networks (such as host addresses and, if enabled, Kubernetes pod CIDRs) are advertised to other cluster members.
Use this setting to prevent certain CIDR ranges from being announced.
Excluded networks will not be reachable over KubeSpan.
**Example:**
```yaml theme={null}
apiVersion: v1alpha1
kind: KubeSpanConfig
filters:
excludeAdvertisedNetworks:
- 172.16.64.0/20
```
In this example, the `172.16.64.0/20` subnet will not be advertised to other cluster members.
> **Note**: Excluded networks must remain reachable via another route (for example, direct L2/L3 routing). Otherwise, they will become unreachable from other nodes.
#### Additional Announced Endpoints
To announce additional endpoints (for example, inbound NAT mappings), use the separate machine configuration document:
```yaml theme={null}
apiVersion: v1alpha1
kind: KubespanEndpointsConfig
extraAnnouncedEndpoints:
- 192.168.101.3:61033
```
## Resource definitions
### KubeSpanIdentities
A node's WireGuard identities can be obtained with:
```sh theme={null}
$ talosctl get kubespanidentities -o yaml
...
spec:
address: fd83:b1f7:fcb5:2802:8c13:71ff:feaf:7c94/128
subnet: fd83:b1f7:fcb5:2802::/64
privateKey: gNoasoKOJzl+/B+uXhvsBVxv81OcVLrlcmQ5jQwZO08=
publicKey: NzW8oeIH5rJyY5lefD9WRoHWWRr/Q6DwsDjMX+xKjT4=
```
Talos automatically configures unique IPv6 address for each node in the cluster-specific IPv6 ULA prefix.
The Wireguard private key is generated and never leaves the node, while the public key is published through the cluster discovery.
`KubeSpanIdentity` is persisted across reboots and upgrades in [STATE](../learn-more/architecture#file-system-partitions) partition in the file `kubespan-identity.yaml`.
### KubeSpanPeerSpecs
A node's WireGuard peers can be obtained with:
```sh theme={null}
$ talosctl get kubespanpeerspecs
ID VERSION LABEL ENDPOINTS
06D9QQOydzKrOL7oeLiqHy9OWE8KtmJzZII2A5/FLFI= 2 talos-default-controlplane-2 ["172.20.0.3:51820"]
THtfKtfNnzJs1nMQKs5IXqK0DFXmM//0WMY+NnaZrhU= 2 talos-default-controlplane-3 ["172.20.0.4:51820"]
nVHu7l13uZyk0AaI1WuzL2/48iG8af4WRv+LWmAax1M= 2 talos-default-worker-2 ["172.20.0.6:51820"]
zXP0QeqRo+CBgDH1uOBiQ8tA+AKEQP9hWkqmkE/oDlc= 2 talos-default-worker-1 ["172.20.0.5:51820"]
```
The peer ID is the Wireguard public key.
`KubeSpanPeerSpecs` are built from the cluster discovery data.
### KubeSpanPeerStatuses
The status of a node's WireGuard peers can be obtained with:
```sh theme={null}
$ talosctl get kubespanpeerstatuses
ID VERSION LABEL ENDPOINT STATE RX TX
06D9QQOydzKrOL7oeLiqHy9OWE8KtmJzZII2A5/FLFI= 63 talos-default-controlplane-2 172.20.0.3:51820 up 15043220 17869488
THtfKtfNnzJs1nMQKs5IXqK0DFXmM//0WMY+NnaZrhU= 62 talos-default-controlplane-3 172.20.0.4:51820 up 14573208 18157680
nVHu7l13uZyk0AaI1WuzL2/48iG8af4WRv+LWmAax1M= 60 talos-default-worker-2 172.20.0.6:51820 up 130072 46888
zXP0QeqRo+CBgDH1uOBiQ8tA+AKEQP9hWkqmkE/oDlc= 60 talos-default-worker-1 172.20.0.5:51820 up 130044 46556
```
KubeSpan peer status includes following information:
* the actual endpoint used for peer communication
* link state:
* `unknown`: the endpoint was just changed, link state is not known yet
* `up`: there is a recent handshake from the peer
* `down`: there is no handshake from the peer
* number of bytes sent/received over the Wireguard link with the peer
If the connection state goes `down`, Talos will be cycling through the available endpoints until it finds the one which works.
Peer status information is updated every 30 seconds.
### KubeSpanEndpoints
A node's WireGuard endpoints (peer addresses) can be obtained with:
```sh theme={null}
$ talosctl get kubespanendpoints
ID VERSION ENDPOINT AFFILIATE ID
06D9QQOydzKrOL7oeLiqHy9OWE8KtmJzZII2A5/FLFI= 1 172.20.0.3:51820 2VfX3nu67ZtZPl57IdJrU87BMjVWkSBJiL9ulP9TCnF
THtfKtfNnzJs1nMQKs5IXqK0DFXmM//0WMY+NnaZrhU= 1 172.20.0.4:51820 b3DebkPaCRLTLLWaeRF1ejGaR0lK3m79jRJcPn0mfA6C
nVHu7l13uZyk0AaI1WuzL2/48iG8af4WRv+LWmAax1M= 1 172.20.0.6:51820 NVtfu1bT1QjhNq5xJFUZl8f8I8LOCnnpGrZfPpdN9WlB
zXP0QeqRo+CBgDH1uOBiQ8tA+AKEQP9hWkqmkE/oDlc= 1 172.20.0.5:51820 6EVq8RHIne03LeZiJ60WsJcoQOtttw1ejvTS6SOBzhUA
```
The endpoint ID is the base64 encoded WireGuard public key.
The observed endpoints are submitted back to the discovery service (if enabled) so that other peers can try additional endpoints to establish the connection.
# Bond
Source: https://docs.siderolabs.com/talos/v1.13/networking/logical/bond
Learn how to configure network bonding.
A bond (also known as link aggregation or NIC teaming) is a method of combining multiple physical network interfaces into a single logical interface to increase bandwidth and provide redundancy.
When one of the physical interfaces in the bond fails, the traffic is automatically redistributed among the remaining interfaces, ensuring continuous network connectivity.
Bond configuration should match the configuration on the connected switch to ensure proper operation.
## Configuration
To configure a bond in Talos Linux, create a [BondConfig](../../reference/configuration/network/bondconfig) document like:
```yaml theme={null}
apiVersion: v1alpha1
kind: BondConfig
name: agg.0
links:
- enp0s2
- enp0s3
bondMode: 802.3ad
miimon: 100
updelay: 200
downdelay: 200
xmitHashPolicy: layer3+4
lacpRate: slow
adActorSysPrio: 65535
resendIGMP: 1
packetsPerSlave: 1
up: true
addresses:
- address: 1.2.3.4/24
```
This example demonstrates how to create a bond named `agg.0` using the interfaces `enp0s2` and `enp0s3` with the 802.3ad (LACP) bonding mode.
The bond is configured with various parameters to optimize performance and reliability, such as `miimon`, `updelay`, `downdelay`, `xmitHashPolicy`, and others.
Addresses and routes can be configured in the same document as shown in [static link configuration](./../configuration/static).
DHCP can be enabled on the bond interface as shown in [dynamic link configuration](./../configuration/dynamic).
The `links` field specifies the physical interfaces that are part of the bond, and [link aliases](./../configuration/aliases) can be used here as well.
Bond MTU can be set using the `mtu` field.
Bond by default inherits MAC address from the first link in the `links` list, but a custom MAC address can be set using the `hardwareAddr` field.
# Bridge
Source: https://docs.siderolabs.com/talos/v1.13/networking/logical/bridge
Learn how to configure network bridges.
A network bridge is a logical network device that connects multiple network interfaces together, allowing them to communicate as if they were on the same physical network segment.
Bridges are commonly used in virtualization and containerization scenarios to allow virtual machines or containers to access the external network through the host machine's network interfaces.
## Configuration
To configure a bridge in Talos Linux, create a [BridgeConfig](../../reference/configuration/network/bridgeconfig) document like:
```yaml theme={null}
apiVersion: v1alpha1
kind: BridgeConfig
name: bridge.1
links:
- eno1
- eno5
stp:
enabled: true
vlan:
filtering: false
up: true
addresses:
- address: 1.2.3.5/32
```
In this example, a bridge named `bridge.1` is created using the physical interfaces `eno1` and `eno5`.
The Spanning Tree Protocol (STP) is enabled to prevent network loops, and VLAN filtering is disabled.
The bridge is also assigned a static IP address of `1.2.3.5/32`.
The `links` field specifies the physical or logical interfaces that are part of the bridge, and [link aliases](./../configuration/aliases) can be used here as well.
Addresses and routes can be configured in the same document as shown in [static link configuration](./../configuration/static).
DHCP can be enabled on the bridge interface as shown in [dynamic link configuration](./../configuration/dynamic).
# VLAN
Source: https://docs.siderolabs.com/talos/v1.13/networking/logical/vlan
Learn how to configure VLANs (virtual LANs).
A VLAN (Virtual Local Area Network) is a logical subdivision of a physical network that allows for the segmentation of network traffic.
VLANs are commonly used to improve network performance and security by isolating different types of traffic within the same physical network infrastructure.
A VLAN can be configured on top of a physical network interface or another logical interface (like a bond).
## Configuration
To configure a VLAN in Talos Linux, create a [VLANConfig](../../reference/configuration/network/vlanconfig) document like:
```yaml theme={null}
apiVersion: v1alpha1
kind: VLANConfig
name: enp0s3.2
vlanID: 2
vlanMode: 802.1q
parent: enp0s3
up: true
addresses:
- address: 192.168.1.100/32
```
In this example, a VLAN with ID `2` is created on top of the physical interface `enp0s3`, and the VLAN interface is named `enp0s3.2`.
The name of the VLAN interface can be customized as needed.
The `vlanMode` field specifies the VLAN tagging protocol to use, with `802.1q` being the most common.
The `parent` field specifies the underlying physical or logical interface on which the VLAN is created.
Addresses and routes can be configured in the same document as shown in [static link configuration](./../configuration/static).
DHCP can be enabled on the VLAN interface as shown in [dynamic link configuration](./../configuration/dynamic).
# Multihoming
Source: https://docs.siderolabs.com/talos/v1.13/networking/multihoming
How to handle multihomed machines
When a machine has multiple IPv4 or IPv6 addresses, it is important to control which addresses Talos components use for communication.
Without explicit configuration, services like etcd or the kubelet may select different addresses across reboots or workloads, leading to unstable networking.
A point to note is that the machines may become multihomed via privileged workloads.
## Multihoming and etcd
The `etcd` cluster needs to establish a mesh of connections among the members.
It is done using the so-called advertised address - each node learns the others’ addresses as they are advertised.
It is crucial that these IP addresses are stable, i.e., that each node always advertises the same IP address.
Moreover, it is beneficial to control them to establish the correct routes between the members and, e.g., avoid congested paths.
In Talos, these addresses are controlled using the `cluster.etcd.advertisedSubnets` configuration key.
## Multihoming and kubelets
Stable IP addressing for kubelets (i.e., nodeIP) is not strictly necessary but highly recommended as it ensures that, e.g., kube-proxy and CNI routing take the desired routes.
Analogously to etcd, for kubelets this is controlled via `machine.kubelet.nodeIP.validSubnets`.
For example, let’s assume that we have a cluster with two networks:
* public network
* private network `192.168.0.0/16`
We want to use the private network for etcd and kubelet communication:
```yaml theme={null}
machine:
kubelet:
nodeIP:
validSubnets:
- 192.168.0.0/16
#...
cluster:
etcd:
advertisedSubnets: # listenSubnets defaults to advertisedSubnets if not set explicitly
- 192.168.0.0/16
```
This way we ensure that the `etcd` cluster will use the private network for communication and the kubelets will use the private network for communication with the control plane.
# Predictable Interface Names
Source: https://docs.siderolabs.com/talos/v1.13/networking/predictable-interface-names
How to use predictable interface naming.
Starting with version Talos 1.5, network interfaces are renamed to [predictable names](https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/)
same way as `systemd` does that in other Linux distributions.
The naming schema `enx78e7d1ea46da` (based on MAC addresses) is enabled by default, the order of interface naming decisions is:
* firmware/BIOS provided index numbers for on-board devices (example: `eno1`)
* firmware/BIOS provided PCI Express hotplug slot index numbers (example: `ens1`)
* physical/geographical location of the connector of the hardware (example: `enp2s0`)
* interfaces's MAC address (example: `enx78e7d1ea46da`)
The predictable network interface names features can be disabled by specifying `net.ifnames=0` in the kernel command line.
Please note that `eth0`, `eth1`, etc. naming scheme is not stable across reboots and can change based on the order of device discovery, so it is not recommended to use it in production environments
when multiple network interfaces are present.
> Note: Talos automatically adds the `net.ifnames=0` kernel argument when upgrading from Talos versions before 1.5, so upgrades to 1.5 don't require any manual intervention.
"Cloud" platforms, like AWS, still use old `eth0` naming scheme as Talos automatically adds `net.ifnames=0` to the kernel command line.
## Single network interface
When running Talos on a machine with a single network interface, predictable interface names might be confusing, as it might come up as `enxSOMETHING` which is hard to address.
There are two ways to solve this:
* disable the feature by supplying `net.ifnames=0` to the initial boot of Talos, Talos will persist `net.ifnames=0` over installs/upgrades.
* use [link aliases](./configuration/aliases):
```yaml theme={null}
apiVersion: v1alpha1
kind: LinkAliasConfig
name: net0
selector:
match: true # as there is a single link only, we can match it this way
```
# SideroLink
Source: https://docs.siderolabs.com/talos/v1.13/networking/siderolink
Point-to-point management overlay Wireguard network.
SideroLink offers a secure point-to-point management overlay network for Talos clusters using Wireguard.
Each Talos machine configured with SideroLink establishes a secure Wireguard connection to the SideroLink API server.
This overlay network utilizes ULA IPv6 addresses, enabling the management of Talos Linux machines even when direct access to their IP addresses is not feasible.
SideroLink is a fundamental component of [Sidero Omni](https://www.siderolabs.com/platform/saas-for-kubernetes/).
## Configuration
To configure SideroLink, provide the SideroLink API server address either via the kernel command line argument `siderolink.api` or as a [config document](../reference/configuration/siderolink/siderolinkconfig).
The SideroLink API URL format is: `https://siderolink.api/?jointoken=token&grpc_tunnel=true`.
* If the URL scheme is `grpc://`, the connection will be established without TLS; otherwise, it will use TLS.
* The join token `token`, if specified, will be sent to the SideroLink server.
* Setting `grpc_tunnel` to `true` sends a hint to tunnel Wireguard traffic over the same SideroLink API gRPC connection instead of using plain UDP.
This is useful in environments where UDP traffic is restricted but adds significant overhead to SideroLink communication, enable this only if necessary.
Note that the SideroLink API server might ignore this hint, and the connection might use gRPC tunneling regardless of the setting.
## Connection flow
1. Talos Linux generates an ephemeral Wireguard key.
2. Talos Linux establishes a gRPC connection to the SideroLink API server, sending its Wireguard public key, join token, and other connection settings.
3. If the join token is valid, the SideroLink API server responds with its Wireguard public key and two overlay IPv6 addresses: one for the machine and one for the SideroLink server.
4. Talos Linux configures the Wireguard interface with the received settings.
5. Talos Linux monitors the Wireguard connection status and re-establishes the connection if necessary.
## Operations with SideroLink
When SideroLink is configured, the Talos maintenance mode API listens exclusively on the SideroLink network.
This allows operations not generally available over the public network, such as retrieving the Talos version and accessing sensitive resources.
Talos Linux always provides the Talos API over SideroLink and automatically permits access over SideroLink even if the [Ingress Firewall](./ingress-firewall) is enabled.
However, Wireguard connections must still be allowed by the Ingress Firewall.
SideroLink only supports point-to-point connections between Talos machines and the SideroLink management server; direct communication between two Talos machines over SideroLink is not possible.
# What is Talos Linux?
Source: https://docs.siderolabs.com/talos/v1.13/overview/what-is-talos
Talos Linux is the best OS for Kubernetes.
Talos Linux is a Kubernetes optimized Linux distro.
It does one thing and it does it better than any general purpose Linux distribution.
Boot it on [bare metal](../getting-started/getting-started) or [in a VM](../platform-specific-installations/cloud-platforms/aws) and have a Kubernetes cluster in minutes.
It runs on your Kubernetes nodes and hosts your workloads.
It also runs the Kubernetes control plane components including the etcd database.
With a single use case and focus it removes complicated and fragile configuration, maintenance, and security vulnerabilities.
It's designed to be as minimal as possible while still maintaining practicality.
For these reasons, Talos has a number of features unique to it:
* API managed
* Immutable file system
* Minimal packages
* Secure by default
Talos can be deployed anywhere you can run a modern Linux kernel.
## API managed
Talos is managed by a single, declarative gRPC API.
This is the most unique thing about Talos and something Talos users love.
Imagine a Linux distribution that had configuration management built in.
That didn't have fragile cloud-init scripts that wrap `bash` scripts in yaml.
Talos ships with a network API for all configuration and troubleshooting needs.
You can [read more about the philosophy behind Talos and Sidero products](../learn-more/philosophy).
## Community & support
Talos has an active and welcoming community. Whether you have a question, found a bug, or want to share feedback, there are several ways to connect:
* **Support**: Ask questions, report bugs, or request features on [GitHub Discussions](https://github.com/siderolabs/talos/discussions)
* **Slack**: Join our [Slack channel](https://taloscommunity.slack.com/). Request access via [inviter.co](https://inviter.co/sidero-labs-community)
* **Twitter**: Follow us at [@SideroLabs](https://twitter.com/SideroLabs)
We welcome users of all experience levels.
### Office hours
We host a recurring community office hours session:
* **When**: Second Monday of every month at 16:30 UTC
* **Where**: [Google Meet](https://meet.google.com/ivb-kjfm-jfc)
Office hours are open to everyone, come ask questions, share feedback, or see what others are building with Talos.
# Air-gapped Environments
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/air-gapped
Setting up Talos Linux to work in environments with no internet access.
When running Talos Linux in air-gapped environments, these are the most common challenges:
* providing access to NTP servers for time synchronization;
* accessing container images, including Talos `installer` images, `etcd`, `kubelet`, Kubernetes control plane images, etc.;
* accessing Image Factory for Talos updates and installation;
* running Discovery Service inside the air-gapped environment.
In this guide, we will assume that the environment is completely air-gapped, with no access to the public Internet.
If there is partial connectivity, most of the requirements can be addresses via [pull-through cache](../configure-your-talos-cluster/images-container-runtime/pull-through-cache) and
HTTP proxy configuration.
## Network configuration
Network configuration in air-gapped environments might require custom settings for DNS and NTP servers.
If running in a virtual environment, the hypervisor might provide time synchronization via [PTP interface](../configure-your-talos-cluster/system-configuration/time-sync) which doesn't require network access.
## Container images
Talos Linux provides support for redirecting image pull requests to internal registries via [registry mirrors](../configure-your-talos-cluster/images-container-runtime/pull-through-cache) feature.
This feature can be used to redirect all image pull requests to an internal registry which is pre-populated with required images.
See the section on [airgapped registry](../configure-your-talos-cluster/images-container-runtime/image-cache-registry-mirror) for more details.
## Image Factory
See the [guide on running Image Factory in air-gapped environments](../../../omni/self-hosted/run-image-factory-on-prem) for more details.
## Discovery service
Talos Linux by default uses the public Discovery Service at `discovery.talos.dev` to facilitate cluster bootstrapping and node discovery.
In air-gapped environments, it is recommended to run a self-hosted instance of the Discovery Service (requires a license from Sidero Labs).
# Boot Loader
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/bare-metal-platforms/bootloader
Overview of the Talos boot process and boot loader configuration.
Talos uses two boot loaders based on system architecture and configuration:
* GRUB
* `systemd-boot`
GRUB is used for legacy BIOS systems on x86\_64, while `systemd-boot` is used for UEFI systems on x86\_64 and arm64.
> Note: When upgrading from earlier Talos versions, the existing bootloader is retained.
> Prior to Talos 1.10, GRUB was the default for all systems except SecureBoot images, which used `systemd-boot`.
To check the current bootloader:
```shell theme={null}
$ talosctl get securitystate -o yaml
spec:
# ...
bootedWithUKI: true # Indicates systemd-boot is in use
```
## `systemd-boot`
`systemd-boot` is the default bootloader for UEFI systems on x86\_64 and arm64.
It is a lightweight boot manager from the `systemd` project, designed for simplicity and speed.
Talos boots via UKI (Unified Kernel Image), a single binary containing the kernel, initramfs, and kernel command line arguments.
The UKI may include multiple profiles with different kernel arguments, such as regular boot and wiping mode.
These profiles are displayed in the `systemd-boot` menu.
Partition layout for `systemd-boot`:
* `EFI`: Contains the `systemd-boot` bootloader and Talos UKIs.
On UEFI systems, the `EFI` partition is automatically detected by the system firmware.
Since UKIs are EFI binaries, they can also be booted directly from the EFI shell or firmware boot menu, including HTTP boot.
With `systemd-boot`, the `.machine.install.extraKernelArgs` field in the machine configuration is ignored, as kernel arguments are embedded in the UKI and cannot be modified without upgrading the UKI.
## GRUB
GRUB boots Talos using `vmlinuz`, `initramfs`, and kernel arguments stored in its configuration file.
> Note: GRUB was previously used for UEFI systems but is no longer used for new installations starting with Talos 1.10.
Partition layout for GRUB:
* MBR (Master Boot Record): Contains the initial boot code.
* `BIOS`: Contains the GRUB bootloader.
* `EFI`: Contains the GRUB bootloader for UEFI systems (only for upgrades from earlier Talos versions).
* `BOOT`: Contains the GRUB configuration file and Talos boot assets (`vmlinuz`, `initramfs`).
With GRUB, kernel arguments are stored in the GRUB configuration file.
The `.machine.install.extraKernelArgs` field in the machine configuration can be used to modify these arguments, followed by an upgrade.
### Controlling kernel command line behavior
Starting from Talos v1.12, you can control how GRUB determines the kernel command line
using the `.machine.install.grubUseUKICmdline` machine configuration option.
When `grubUseUKICmdline` is set to `true`, GRUB will use the kernel command line
embedded in the Unified Kernel Image (UKI) that was generated during Talos image creation.
This behavior makes it easier to customize kernel parameters when generating boot assets.
When set to `false`, GRUB will ignore the UKI-provided kernel command line and instead
construct its own command line based on Talos defaults and configuration (the legacy behavior).
For existing installations upgrading to v1.12, this option will default to false to preserve the legacy behavior.
# Equinix Metal
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/bare-metal-platforms/equinix-metal
Creating Talos clusters with Equinix Metal.
You can create a Talos Linux cluster on Equinix Metal in a variety of ways, such as through the EM web UI, or the `metal` command line tool.
Regardless of the method, the process is:
* Create a DNS entry for your Kubernetes endpoint.
* Generate the configurations using `talosctl`.
* Provision your machines on Equinix Metal.
* Push the configurations to your servers (if not done as part of the machine provisioning).
* Configure your Kubernetes endpoint to point to the newly created control plane nodes.
* Bootstrap the cluster.
## Define the Kubernetes endpoint
There are a variety of ways to create an HA endpoint for the Kubernetes cluster.
Some of the ways are:
* DNS
* Load Balancer
* BGP
Whatever way is chosen, it should result in an IP address/DNS name that routes traffic to all the control plane nodes.
We do not know the control plane node IP addresses at this stage, but we should define the endpoint DNS entry so that we can use it in creating the cluster configuration.
After the nodes are provisioned, we can use their addresses to create the endpoint A records, or bind them to the load balancer, etc.
## Create the machine configuration files
### Generating configurations
Using the DNS name of the loadbalancer defined above, generate the base configuration files for the Talos machines:
```bash theme={null}
$ talosctl gen config talos-k8s-em-tutorial https://:
created controlplane.yaml
created worker.yaml
created talosconfig
```
> The `port` used above should be 6443, unless your load balancer maps a different port to port 6443 on the control plane nodes.
### Validate the configuration files
```bash theme={null}
talosctl validate --config controlplane.yaml --mode metal
talosctl validate --config worker.yaml --mode metal
```
> Note: Validation of the install disk could potentially fail as validation
> is performed on your local machine and the specified disk may not exist.
### Passing in the configuration as user data
You can use the metadata service provide by Equinix Metal to pass in the machines configuration.
It is required to add a shebang to the top of the configuration file.
The convention we use is `#!talos`.
## Provision the machines in Equinix Metal
Talos Linux can be PXE-booted on Equinix Metal using [Image Factory](../../learn-more/image-factory), using the `equinixMetal` platform: e.g.
This URL references the default schematic for Talos version with an `amd64` architecture.
Follow the Image Factory guide to create a custom schematic, e.g. with CPU microcode updates.
The PXE boot URL can be used as the iPXE script URL.
### Using the Equinix Metal UI
Simply select the location and type of machines in the Equinix Metal web interface.
Select 'Custom iPXE' as the Operating System and enter the Image Factory PXE URL as the iPXE script URL, then select the number of servers to create, and name them (in lowercase only.)
Under *optional settings*, you can optionally paste in the contents of `controlplane.yaml` that was generated, above (ensuring you add a first line of `#!talos`).
You can repeat this process to create machines of different types for control plane and worker nodes (although you would pass in `worker.yaml` for the worker nodes, as user data).
If you did not pass in the machine configuration as User Data, you need to provide it to each machine, with the following command:
`talosctl apply-config --insecure --nodes --file ./controlplane.yaml`
### Creating a cluster via the Equinix Metal CLI
This guide assumes the user has a working API token,and the [Equinix Metal CLI](https://github.com/equinix/metal-cli/) installed.
> Note: Ensure you have prepended `#!talos` to the `controlplane.yaml` file.
e.g.
Repeat this to create each control plane node desired: there should usually be 3 for a HA cluster.
## Update the Kubernetes endpoint
Now our control plane nodes have been created, and we know their IP addresses, we can associate them with the Kubernetes endpoint.
Configure your load balancer to route traffic to these nodes, or add `A` records to your DNS entry for the endpoint, for each control plane node.
e.g.
```bash theme={null}
host endpoint.mydomain.com
endpoint.mydomain.com has address 145.40.90.201
endpoint.mydomain.com has address 147.75.109.71
endpoint.mydomain.com has address 145.40.90.177
```
## Bootstrap etcd
Set the `endpoints` and `nodes` for `talosctl`:
```bash theme={null}
talosctl --talosconfig talosconfig config endpoint
talosctl --talosconfig talosconfig config node
```
Bootstrap `etcd`:
```bash theme={null}
talosctl --talosconfig talosconfig bootstrap
```
This only needs to be issued to one control plane node.
## Retrieve the `kubeconfig`
At this point we can retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl --talosconfig talosconfig kubeconfig .
```
# ISO
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/bare-metal-platforms/iso
Booting Talos on bare-metal with ISO.
Talos can be installed on bare-metal machine using an ISO image.
ISO images for `amd64` and `arm64` architectures are available on the [Talos releases page](https://github.com/siderolabs/talos/releases/latest/).
Talos doesn't install itself to disk when booted from an ISO until the machine configuration is applied.
Please follow the [getting started guide](../../getting-started/getting-started) for the generic steps on how to install Talos.
> Note: If there is already a Talos installation on the disk, the machine will boot into that installation when booting from a Talos ISO.
> The boot order should prefer disk over ISO, or the ISO should be removed after the installation to make Talos boot from disk.
See [kernel parameters reference](../../reference/kernel) for the list of kernel parameters supported by Talos.
There are two flavors of ISO images available:
* `metal-.iso` supports booting on BIOS and UEFI systems (for x86, UEFI only for arm64)
* `metal--secureboot.iso` supports booting on only UEFI systems in SecureBoot mode (via [Image Factory](../../learn-more/image-factory))
# Matchbox
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/bare-metal-platforms/matchbox
In this guide we will create an HA Kubernetes cluster with 3 worker nodes using an existing load balancer and matchbox deployment.
## Creating a cluster
In this guide we will create an HA Kubernetes cluster with 3 worker nodes.
We assume an existing load balancer, matchbox deployment, and some familiarity with iPXE.
We leave it up to the user to decide if they would like to use static networking, or DHCP.
The setup and configuration of DHCP will not be covered.
### Create the machine configuration files
#### Generating base configurations
Using the DNS name of the load balancer, generate the base configuration files for the Talos machines:
```bash theme={null}
$ talosctl gen config talos-k8s-metal-tutorial https://:
created controlplane.yaml
created worker.yaml
created talosconfig
```
At this point, you can modify the generated configs to your liking.
Optionally, you can specify [machine configuration patches](../../configure-your-talos-cluster/system-configuration/patching#configuration-patching-with-talosctl-cli) which will be applied during the config generation.
#### Validate the configuration files
```bash theme={null}
$ talosctl validate --config controlplane.yaml --mode metal
controlplane.yaml is valid for metal mode
$ talosctl validate --config worker.yaml --mode metal
worker.yaml is valid for metal mode
```
#### Publishing the machine configuration files
In bare-metal setups it is up to the user to provide the configuration files over HTTP(S).
A special kernel parameter (`talos.config`) must be used to inform Talos about *where* it should retrieve its configuration file.
To keep things simple we will place `controlplane.yaml`, and `worker.yaml` into Matchbox's `assets` directory.
This directory is automatically served by Matchbox.
### Create the Matchbox configuration files
The profiles we will create will reference `vmlinuz`, and `initramfs.xz`.
Download these files from the [release](https://github.com/siderolabs/talos/releases) of your choice, and place them in `/var/lib/matchbox/assets`.
#### Profiles
##### Control plane nodes
```json theme={null}
{
"id": "control-plane",
"name": "control-plane",
"boot": {
"kernel": "/assets/vmlinuz",
"initrd": ["/assets/initramfs.xz"],
"args": [
"initrd=initramfs.xz",
"init_on_alloc=1",
"slab_nomerge",
"pti=on",
"console=tty0",
"printk.devkmsg=on",
"talos.platform=metal",
"talos.config=http://matchbox.talos.dev/assets/controlplane.yaml"
]
}
}
```
> Note: Be sure to change `http://matchbox.talos.dev` to the endpoint of your matchbox server.
##### Worker nodes
```json theme={null}
{
"id": "default",
"name": "default",
"boot": {
"kernel": "/assets/vmlinuz",
"initrd": ["/assets/initramfs.xz"],
"args": [
"initrd=initramfs.xz",
"init_on_alloc=1",
"slab_nomerge",
"pti=on",
"console=tty0",
"printk.devkmsg=on",
"talos.platform=metal",
"talos.config=http://matchbox.talos.dev/assets/worker.yaml"
]
}
}
```
#### Groups
Now, create the following groups, and ensure that the `selector`s are accurate for your specific setup.
```json theme={null}
{
"id": "control-plane-1",
"name": "control-plane-1",
"profile": "control-plane",
"selector": {
...
}
}
```
```json theme={null}
{
"id": "control-plane-2",
"name": "control-plane-2",
"profile": "control-plane",
"selector": {
...
}
}
```
```json theme={null}
{
"id": "control-plane-3",
"name": "control-plane-3",
"profile": "control-plane",
"selector": {
...
}
}
```
```json theme={null}
{
"id": "default",
"name": "default",
"profile": "default"
}
```
### Boot the machines
Now that we have our configuration files in place, boot all the machines.
Talos will come up on each machine, grab its configuration file, and bootstrap itself.
### Bootstrap etcd
Set the `endpoints` and `nodes`:
```bash theme={null}
talosctl --talosconfig talosconfig config endpoint
talosctl --talosconfig talosconfig config node
```
Bootstrap `etcd`:
```bash theme={null}
talosctl --talosconfig talosconfig bootstrap
```
### Retrieve the `kubeconfig`
At this point we can retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl --talosconfig talosconfig kubeconfig .
```
# Metal Network Configuration
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/bare-metal-platforms/metal-network-configuration
How to use `META`-based network configuration on Talos `metal` platform.
> Note: This is an advanced feature which requires deep understanding of Talos and Linux network configuration.
> Note: It is recommended to use Talos machine configuration for network configuration, possibly embedded into the image itself,
> to avoid the complexity of `META`-based network configuration.
Talos Linux when running on a cloud platform (e.g. AWS or Azure), uses the platform-provided metadata server to provide initial network configuration to the node.
When running on bare-metal, there is no metadata server, so there are several options to provide initial network configuration (before machine configuration is acquired):
* use automatic network configuration via DHCP (Talos default)
* use initial boot [kernel command line parameters](../../reference/kernel) to configure networking
* use automatic network configuration via DHCP just enough to fetch machine configuration and then use machine configuration to set desired advanced configuration.
If DHCP option is available, it is by far the easiest way to configure networking.
The initial boot kernel command line parameters are not very flexible, and they are not persisted after initial Talos installation.
Talos starting with version 1.4.0 offers a new option to configure networking on bare-metal: `META`-based network configuration.
> Note: `META`-based network configuration is only available on Talos Linux `metal` platform.
Talos [dashboard](../../deploy-and-manage-workloads/interactive-dashboard) provides a way to configure `META`-based network configuration for a machine using the console, but
it doesn't support all kinds of network configuration.
## Network configuration format
Talos `META`-based network configuration is a YAML file with the following format:
```yaml theme={null}
addresses:
- address: 147.75.61.43/31
linkName: bond0
family: inet4
scope: global
flags: permanent
layer: platform
- address: 2604:1380:45f2:6c00::1/127
linkName: bond0
family: inet6
scope: global
flags: permanent
layer: platform
- address: 10.68.182.1/31
linkName: bond0
family: inet4
scope: global
flags: permanent
layer: platform
links:
- name: eth0
up: true
masterName: bond0
slaveIndex: 0
layer: platform
- name: eth1
up: true
masterName: bond0
slaveIndex: 1
layer: platform
- name: bond0
logical: true
up: true
mtu: 0
kind: bond
type: ether
bondMaster:
mode: 802.3ad
xmitHashPolicy: layer3+4
lacpRate: slow
arpValidate: none
arpAllTargets: any
primaryReselect: always
failOverMac: 0
miimon: 100
updelay: 200
downdelay: 200
resendIgmp: 1
lpInterval: 1
packetsPerSlave: 1
numPeerNotif: 1
tlbLogicalLb: 1
adActorSysPrio: 65535
layer: platform
routes:
- family: inet4
gateway: 147.75.61.42
outLinkName: bond0
table: main
priority: 1024
scope: global
type: unicast
protocol: static
layer: platform
- family: inet6
gateway: '2604:1380:45f2:6c00::'
outLinkName: bond0
table: main
priority: 2048
scope: global
type: unicast
protocol: static
layer: platform
- family: inet4
dst: 10.0.0.0/8
gateway: 10.68.182.0
outLinkName: bond0
table: main
scope: global
type: unicast
protocol: static
layer: platform
hostnames:
- hostname: ci-blue-worker-amd64-2
layer: platform
resolvers: []
timeServers: []
```
Every section is optional, so you can configure only the parts you need.
The format of each section matches the respective network [`*Spec` resource](../../learn-more/networking-resources) `.spec` part, e.g the `addresses:`
section matches the `.spec` of `AddressSpec` resource:
```yaml theme={null}
# talosctl get addressspecs bond0/10.68.182.1/31 -o yaml | yq .spec
address: 10.68.182.1/31
linkName: bond0
family: inet4
scope: global
flags: permanent
layer: platform
```
So one way to prepare the network configuration file is to boot Talos Linux, apply necessary network configuration using Talos machine configuration, and grab the resulting
resources from the running Talos instance.
In this guide we will briefly cover the most common examples of the network configuration.
### Addresses
The addresses configured are usually routable IP addresses assigned to the machine, so
the `scope:` should be set to `global` and `flags:` to `permanent`.
Additionally, `family:` should be set to either `inet4` or `inet6` depending on the address family.
The `linkName:` property should match the name of the link the address is assigned to, it might be a physical link,
e.g. `en9sp0`, or the name of a logical link, e.g. `bond0`, created in the `links:` section.
Example, IPv4 address:
```yaml theme={null}
addresses:
- address: 147.75.61.43/31
linkName: bond0
family: inet4
scope: global
flags: permanent
layer: platform
```
Example, IPv6 address:
```yaml theme={null}
addresses:
- address: 2604:1380:45f2:6c00::1/127
linkName: bond0
family: inet6
scope: global
flags: permanent
layer: platform
```
### Links
For physical network interfaces (links), the most usual configuration is to bring the link up:
```yaml theme={null}
links:
- name: en9sp0
up: true
layer: platform
```
This will bring the link up, and it will also disable Talos auto-configuration (disables running DHCP on the link).
Another common case is to set a custom MTU:
```yaml theme={null}
links:
- name: en9sp0
up: true
mtu: 9000
layer: platform
```
The order of the links in the `links:` section is not important.
#### Bonds
For bonded links, there should be a link resource for the bond itself, and a link resource for each enslaved link:
```yaml theme={null}
links:
- name: bond0
logical: true
up: true
kind: bond
type: ether
bondMaster:
mode: 802.3ad
xmitHashPolicy: layer3+4
lacpRate: slow
arpValidate: none
arpAllTargets: any
primaryReselect: always
failOverMac: 0
miimon: 100
updelay: 200
downdelay: 200
resendIgmp: 1
lpInterval: 1
packetsPerSlave: 1
numPeerNotif: 1
tlbLogicalLb: 1
adActorSysPrio: 65535
layer: platform
- name: eth0
up: true
masterName: bond0
slaveIndex: 0
layer: platform
- name: eth1
up: true
masterName: bond0
slaveIndex: 1
layer: platform
```
The name of the bond can be anything supported by Linux kernel, but the following properties are important:
* `logical: true` - this is a logical link, not a physical one
* `kind: bond` - this is a bonded link
* `type: ether` - this is an Ethernet link
* `bondMaster:` - defines bond configuration, please see Linux documentation on the available options
For each enslaved link, the following properties are important:
* `masterName: bond0` - the name of the bond this link is enslaved to
* `slaveIndex: 0` - the index of the enslaved link, starting from 0, controls the order of bond slaves
#### VLANs
VLANs are logical links which have a parent link, and a VLAN ID and protocol:
```yaml theme={null}
links:
- name: bond0.35
logical: true
up: true
kind: vlan
type: ether
parentName: bond0
vlan:
vlanID: 35
vlanProtocol: 802.1ad
```
The name of the VLAN link can be anything supported by Linux kernel, but the following properties are important:
* `logical: true` - this is a logical link, not a physical one
* `kind: vlan` - this is a VLAN link
* `type: ether` - this is an Ethernet link
* `parentName: bond0` - the name of the parent link
* `vlan:` - defines VLAN configuration: `vlanID` and `vlanProtocol`
### Routes
For route configuration, most of the time `table: main`, `scope: global`, `type: unicast` and `protocol: static` are used.
The route most important fields are:
* `dst:` defines the destination network, if left empty means "default gateway"
* `gateway:` defines the gateway address
* `priority:` defines the route priority (metric), lower values are preferred for the same `dst:` network
* `outLinkName:` defines the name of the link the route is associated with
* `src:` sets the source address for the route (optional)
Additionally, `family:` should be set to either `inet4` or `inet6` depending on the address family.
Example, IPv6 default gateway:
```yaml theme={null}
routes:
- family: inet6
gateway: '2604:1380:45f2:6c00::'
outLinkName: bond0
table: main
priority: 2048
scope: global
type: unicast
protocol: static
layer: platform
```
Example, IPv4 route to `10/8` via `10.68.182.0` gateway:
```yaml theme={null}
routes:
- family: inet4
dst: 10.0.0.0/8
gateway: 10.68.182.0
outLinkName: bond0
table: main
scope: global
type: unicast
protocol: static
layer: platform
```
### Hostnames
Even though the section supports multiple hostnames, only a single one should be used:
```yaml theme={null}
hostnames:
- hostname: host
domainname: some.org
layer: platform
```
The `domainname:` is optional.
If the hostname is not set, Talos will use default generated hostname.
### Resolvers
The `resolvers:` section is used to configure DNS resolvers, only single entry should be used:
```yaml theme={null}
resolvers:
- dnsServers:
- 8.8.8.8
- 1.1.1.1
layer: platform
```
If the `dnsServers:` is not set, Talos will use default DNS servers.
### Time Servers
The `timeServers:` section is used to configure NTP time servers, only single entry should be used:
```yaml theme={null}
timeServers:
- timeServers:
- 169.254.169.254
layer: platform
```
If the `timeServers:` is not set, Talos will use default NTP servers.
## Supplying `META` network configuration
Once the network configuration YAML document is ready, it can be supplied to Talos in one of the following ways:
* for a running Talos machine, using Talos API (requires already established network connectivity)
* for Talos disk images, it can be embedded into the image
* for ISO/PXE boot methods, it can be supplied via kernel command line parameters as an environment variable
The metal network configuration is stored in Talos `META` partition under the key `0xa` (decimal 10).
In this guide we will assume that the prepared network configuration is stored in the file `network.yaml`.
> Note: as JSON is a subset of YAML, the network configuration can be also supplied as a JSON document.
### Supplying network configuration to a running Talos machine
Use the `talosctl` to write a network configuration to a running Talos machine:
```bash theme={null}
talosctl meta write 0xa "$(cat network.yaml)"
```
### Supplying network configuration to a Talos disk image
Following the [boot assets](../../platform-specific-installations/boot-assets) guide, create a disk image passing the network configuration as a `--meta` flag:
### Supplying network configuration to a Talos ISO/PXE boot
As there is no `META` partition created yet before Talos Linux is installed, `META` values can be set as an environment variable `INSTALLER_META_BASE64` passed to the initial boot of Talos.
The supplied value will be used immediately, and also it will be written to the `META` partition once Talos is installed.
When using `imager` to create the ISO, the `INSTALLER_META_BASE64` environment variable will be automatically generated from the `--meta` flag:
When PXE booting, the value of `INSTALLER_META_BASE64` should be set manually:
```bash theme={null}
echo -n "0xa=$(cat network.yaml)" | gzip -9 | base64
```
The resulting base64 string should be passed as an environment variable `INSTALLER_META_BASE64` to the initial boot of Talos: `talos.environment=INSTALLER_META_BASE64=`.
### Getting current `META` network configuration
Talos exports `META` keys as resources:
```yaml theme={null}
# talosctl get meta 0x0a -o yaml
...
spec:
value: '{"addresses": ...}'
```
# Network Configuration
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/bare-metal-platforms/network-config
In this guide we will describe how network can be configured on bare-metal platforms.
By default, Talos will run DHCP client on all interfaces which have a link, and that might be enough for most of the cases.
If some advanced network configuration is required, it can be done via the [machine configuration](../../networking/configuration/overview) file.
But sometimes it is required to apply network configuration even before the machine configuration can be fetched from the network.
In this case, the recommended way is to embed the machine configuration into the Talos image itself, as described in the [boot assets guide](../../platform-specific-installations/boot-assets).
If machine configuration embedding is not possible, Talos provides several ways to configure network on bare-metal platforms before the machine configuration is fetched.
## Kernel command line
Talos supports some kernel command line parameters to configure network before the machine configuration is fetched.
> Note: Kernel command line parameters are not persisted after Talos installation, so proper network configuration should be done via the machine configuration.
Address, default gateway and DNS servers can be configured via `ip=` kernel command line parameter:
```text theme={null}
ip=172.20.0.2::172.20.0.1:255.255.255.0::eth0.100:::::
```
Bonding can be configured via `bond=` kernel command line parameter:
```text theme={null}
bond=bond0:eth0,eth1:balance-rr
```
VLANs can be configured via `vlan=` kernel command line parameter:
```text theme={null}
vlan=eth0.100:eth0
```
See [kernel parameters reference](../../reference/kernel) for more details.
## Platform network configuration
Some platforms (e.g. AWS, Google Cloud, etc.) have their own network configuration mechanisms, which can be used to perform the initial network configuration.
There is no such mechanism for bare-metal platforms, so Talos provides a way to use platform network config on the `metal` platform to submit the initial network configuration.
The platform network configuration is a YAML document which contains resource specifications for various network resources.
For the `metal` platform, the [interactive dashboard](../../deploy-and-manage-workloads/interactive-dashboard) can be used to edit the platform network configuration, also the configuration can be
created [manually](./metal-network-configuration).
The current value of the platform network configuration can be retrieved using the `MetaKeys` resource (key `0x0a`):
```bash theme={null}
talosctl get meta 0x0a
```
The platform network configuration can be updated using the `talosctl meta` command for the running node:
```bash theme={null}
talosctl meta write 0x0a '{"externalIPs": ["1.2.3.4"]}'
talosctl meta delete 0x0a
```
The initial platform network configuration for the `metal` platform can be also included into the generated Talos image:
The platform network configuration gets merged with other sources of network configuration, the details can be found in the [network resources guide](../../learn-more/networking-resources#configuration-merging).
## `nocloud` network configuration
Some bare-metal providers provide a way to configure network via the `nocloud` data source.
Talos Linux can automatically pick up this [configuration](../cloud-platforms/nocloud) when `nocloud` image is used.
# PXE
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/bare-metal-platforms/pxe
Booting Talos over the network on bare-metal with PXE.
Talos can be installed on bare-metal using PXE service.
There are more detailed guides for PXE booting using [Matchbox](./matchbox).
This guide describes generic steps for PXE booting Talos on bare-metal.
First, download the `vmlinuz` and `initramfs` assets from the [Talos releases page](https://github.com/siderolabs/talos/releases/latest/).
Set up the machines to PXE boot from the network (usually by setting the boot order in the BIOS).
There might be options specific to the hardware being used, booting in BIOS or UEFI mode, using iPXE, etc.
Talos requires the following kernel parameters to be set on the initial boot:
* `talos.platform=metal`
* `slab_nomerge`
* `pti=on`
When booted from the network without machine configuration, Talos will start in maintenance mode.
Please follow the [getting started guide](../../getting-started/getting-started) for the generic steps on how to install Talos.
See [kernel parameters reference](../../reference/kernel) for the list of kernel parameters supported by Talos.
> Note: If there is already a Talos installation on the disk, the machine will boot into that installation when booting from network.
> The boot order should prefer disk over network.
Talos can automatically fetch the machine configuration from the network on the initial boot using `talos.config` kernel parameter.
A metadata service (HTTP service) can be implemented to deliver customized configuration to each node for example by using the MAC address of the node:
```text theme={null}
talos.config=https://metadata.service/talos/config?mac=${mac}
```
> Note: The `talos.config` kernel parameter supports other substitution variables, see [kernel parameters reference](../../reference/kernel) for the full list.
PXE booting can be also performed via [Image Factory](../../learn-more/image-factory).
# SecureBoot
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/bare-metal-platforms/secureboot
Booting Talos in SecureBoot mode on UEFI platforms.
Talos supports booting on UEFI systems in SecureBoot mode.
When combined with TPM-based disk encryption, this provides a complete [Trusted Boot](https://0pointer.net/blog/brave-new-trusted-boot-world.html) experience where the entire boot chain is cryptographically verified.
This means the disk will only unlock if SecureBoot remains enabled with the same key set when using the default PCR 7 binding. However, **PCR binding is fully configurable** via the `VolumeConfig` `tpm.pcrs` option - see the [TPM encryption options](../../reference/configuration/block/volumeconfig#VolumeConfig.encryption.keys..tpm) for details.
## PCR binding options
* **Default**: PCR 7 (SecureBoot state) + PCR 11 signed policy (UKI measurements and boot phases)
* **Configurable**: Any combination of PCRs can be specified
* **No PCRs**: Can be disabled by passing an empty list, relying solely on PCR 11 signed policy
* **Backward compatibility**: Existing installations continue to use their original PCR binding
**Why configurable PCRs?**
* **Frequent Updates**: PCR 7 covers the SecureBoot policy, particularly the "dbx" denylist of revoked certificates
* **Automatic Updates**: Tools like `fwupd` now automatically update the SecureBoot database, causing PCR 7 to change frequently
* **Brittleness**: Literal PCR 7 policies break when firmware or SecureBoot databases are updated, even without policy changes
Talos maintains PCR 7 binding by default for backward compatibility, but users can now choose configurations that better suit their update policies and security requirements.
When the UKI image is generated, the UKI is measured and expected measurements are combined into TPM unlock policy and signed with the PCR signing key. This provides [Trusted Boot](https://0pointer.net/blog/brave-new-trusted-boot-world.html) experience.
> Note: SecureBoot is not supported on x86 platforms in BIOS mode.
## SecureBoot flow
The SecureBoot process follows a strict verification chain from UEFI firmware to the final operating system:
```mermaid theme={null}
flowchart TD
UEFI[UEFI Firmware]
KEYS{SecureBoot Keys
Enrolled?}
SETUP[Setup Mode
Auto-enroll Keys]
SYSTEMD[systemd-boot
Bootloader]
UKI[Unified Kernel Image
UKI]
MEASURE[TPM PCR
Measurements]
DISK{Encrypted
Disk?}
UNSEAL[TPM Unseal
Disk Key]
POLICY[Verify PCR Policy
+ Signature]
TALOS[Talos Linux
Operating System]
FAIL[Boot Failure]
UEFI --> KEYS
KEYS -->|No| SETUP
SETUP --> SYSTEMD
KEYS -->|Yes| SYSTEMD
SYSTEMD -->|Verify Signature| UKI
SYSTEMD -->|Invalid Signature| FAIL
UKI --> MEASURE
MEASURE --> DISK
DISK -->|Yes| POLICY
POLICY -->|Valid Policy
+ PCR State| UNSEAL
POLICY -->|Invalid| FAIL
UNSEAL -->|Success| TALOS
UNSEAL -->|Failure| FAIL
DISK -->|No| TALOS
style UEFI fill:#e1f5fe
style UKI fill:#f3e5f5
style MEASURE fill:#fff3e0
style POLICY fill:#e8f5e8
style FAIL fill:#ffebee
style TALOS fill:#e0f2f1
```
The implementation is using [systemd-boot](https://www.freedesktop.org/wiki/Software/systemd/systemd-boot/) as a boot menu implementation, while the
Talos kernel, initramfs and cmdline arguments are combined into the [Unified Kernel Image](https://uapi-group.org/specifications/specs/unified_kernel_image/) (UKI) format.
UEFI firmware loads the `systemd-boot` bootloader, which then loads the UKI image.
Both `systemd-boot` and Talos `UKI` image are signed with the key, which is enrolled into the UEFI firmware.
As Talos Linux is fully contained in the UKI image, the full operating system is verified and booted by the UEFI firmware.
> Note: There is no support at the moment to upgrade non-UKI (GRUB-based) Talos installation to use UKI/SecureBoot, so a fresh installation is required.
## SecureBoot with Sidero Labs images
[Sidero Labs](https://www.siderolabs.com/) provides Talos images signed with the [Sidero Labs SecureBoot key](https://factory.talos.dev/secureboot/signing-cert.pem) via [Image Factory](../../learn-more/image-factory).
> Note: The SecureBoot images are available for Talos releases starting from `v1.5.0`.
The easiest way to get started with SecureBoot is to download the ISO, and
boot it on a UEFI-enabled system which has SecureBoot enabled in setup mode.
The ISO bootloader will enroll the keys in the UEFI firmware, and boot the Talos Linux in SecureBoot mode.
The install should be performed using SecureBoot installer (put it Talos machine configuration):
> Note: SecureBoot images can also be generated with [custom keys](#secureboot-with-custom-keys).
## Booting Talos Linux in SecureBoot mode
In this guide we will use the ISO image to boot Talos Linux in SecureBoot mode, followed by submitting machine configuration to the machine in maintenance mode.
We will use one the ways to generate and submit machine configuration to the node, please refer to the [Production Notes](../../getting-started/prodnotes) for the full guide.
First, make sure SecureBoot is enabled in the UEFI firmware.
For the first boot, the UEFI firmware should be in the setup mode, so that the keys can be enrolled into the UEFI firmware automatically.
If the UEFI firmware does not support automatic enrollment, you may need to hit Esc to force the boot menu to appear, and select the `Enroll Secure Boot keys: auto` option.
> Note: There are other ways to enroll the keys into the UEFI firmware, but this is out of scope of this guide.
Once Talos is running in maintenance mode, verify that secure boot is enabled:
```shell theme={null}
$ talosctl -n get securitystate --insecure
NODE NAMESPACE TYPE ID VERSION SECUREBOOT
runtime SecurityState securitystate 1 true
```
Now we will generate the machine configuration for the node supplying the `installer-secureboot` container image, and applying the patch to enable TPM-based [disk encryption](../../configure-your-talos-cluster/storage-and-disk-management/disk-encryption) (requires TPM 2.0):
```yaml theme={null}
# tpm-disk-encryption.yaml
machine:
systemDiskEncryption:
ephemeral:
provider: luks2
keys:
- slot: 0
tpm: {}
state:
provider: luks2
keys:
- slot: 0
tpm: {}
```
Generate machine configuration:
Apply machine configuration to the node:
```shell theme={null}
talosctl -n apply-config --insecure -f controlplane.yaml
```
Talos will perform the installation to the disk and reboot the node.
Please make sure that the ISO image is not attached to the node anymore, otherwise the node will boot from the ISO image again.
Once the node is rebooted, verify that the node is running in secure boot mode:
```shell theme={null}
talosctl -n --talosconfig=talosconfig get securitystate
```
## Upgrading Talos Linux
Any change to the boot asset (kernel, initramfs, kernel command line) requires the UKI to be regenerated and the installer image to be rebuilt.
Follow the steps above to generate new installer image updating the boot assets: use new Talos version, add a system extension, or modify the kernel command line.
Once the new `installer` image is pushed to the registry, [upgrade](../../configure-your-talos-cluster/lifecycle-management/upgrading-talos) the node using the new installer image.
It is important to preserve the UKI signing key and the PCR signing key, otherwise the node will not be able to boot with the new UKI and unlock the encrypted partitions.
## Disk encryption with TPM
When encrypting the disk partition for the first time, Talos Linux generates a random disk encryption key and seals (encrypts) it with the TPM device.
The TPM unlock policy is configured to trust the expected policy signed by the PCR signing key.
This way TPM unlocking doesn't depend on the exact [PCR measurements](https://uapi-group.org/specifications/specs/linux_tpm_pcr_registry/), but rather on the expected policy signed by the PCR signing key and the configured PCR states (by default includes PCR 7 for SecureBoot status and the list of enrolled keys, plus PCR 11 for boot integrity).
### PCR measurements in detail
The Unified Kernel Image (UKI) boot process involves several measurement stages that record cryptographic hashes into TPM Platform Configuration Registers (PCRs):
#### systemd-stub UKI measurements (PCR 11)
According to the [UAPI Unified Kernel Image specification](https://uapi-group.org/specifications/specs/unified_kernel_image/) and [systemd-stub documentation](https://www.freedesktop.org/software/systemd/man/latest/systemd-stub.html), systemd-stub measures the following UKI sections into **PCR 11**:
* **`.linux` section** - The Linux kernel binary (PE section containing the ELF kernel image, required)
* **`.osrel` section** - OS release information (PE section with `/etc/os-release` contents)
* **`.cmdline` section** - The kernel command line arguments (PE section with embedded cmdline)
* **`.initrd` section** - The initial ramdisk image (PE section containing initramfs)
* **`.ucode` section** - Microcode initrd (PE section with CPU microcode updates, uncompressed)
* **`.splash` section** - Boot splash image (PE section with Windows BMP format image)
* **`.dtb` section** - Device tree blob (PE section with compiled binary DeviceTree)
* **`.dtbauto` sections** - Automatic DeviceTree selection (zero or more PE sections, first match used)
* **`.efifw` sections** - Firmware images (zero or more PE sections for firmware blobs)
* **`.hwids` sections** - Hardware ID matching (zero or more PE sections with SMBIOS-based hardware IDs)
* **`.uname` section** - Kernel version information (PE section with `uname -r` output)
* **`.sbat` section** - SBAT revocation metadata (PE section for Secure Boot Advanced Targeting)
* **`.pcrpkey` section** - PCR signature public key (PE section with PEM format public key)
**Note:** The `.pcrsig` section is **not measured** into any PCR, as it contains the signatures for the measurement results themselves.
#### systemd-boot Measurements
The [systemd-boot bootloader](https://www.freedesktop.org/software/systemd/man/latest/systemd-boot.html) can optionally measure loaded boot entries and configuration, though this is typically not used in Talos UKI scenarios since the UKI can be loaded directly.
#### Talos Boot phase measurements (PCR 11)
In addition to the UKI section measurements, Talos extends **PCR 11** with its own boot phases to track the operating system initialization:
1. **`enter-initrd`** - Extended when Talos initrd starts
2. **`leave-initrd`** - Extended just before switching to the main system (machined)
3. **`enter-machined`** - Extended before starting the main Talos supervisor
4. **`start-the-world`** - Extended **after disk decryption** and before starting all system services
**Important:** The `start-the-world` phase is measured into PCR 11 *after* the encrypted disk has been unlocked. This ensures that user services and workloads cannot decrypt the disk themselves, as any attempt to access TPM-sealed keys will fail due to the changed PCR 11 value.
#### TPM unlock policy
The TPM sealed disk encryption key can only be unsealed when the system reaches the **`enter-machined`** phase. This is the critical security boundary - the disk can only be decrypted if:
* The UKI sections (kernel, initrd, cmdline, etc.) match the expected measurements (PCR 11)
* The boot reached the legitimate `enter-machined` phase (PCR 11)
* The configured PCR states match (by default, includes PCR 7 for SecureBoot state)
This ensures that disk decryption only occurs after the trusted boot chain has been verified, but before any potentially untrusted user workloads start.
#### Configurable PCR binding (Default: PCR 7)
By default, new Talos installations and upgrades maintain binding to **PCR 7**, which includes:
* SecureBoot enabled/disabled state
* Enrolled SecureBoot keys (PK, KEK, db)
* Any changes to the UEFI SecureBoot configuration
This means the disk will only unlock if SecureBoot remains enabled with the same key set. However, this PCR 7 binding can be optionally disabled via the `VolumeConfig` `tpm.pcrs` option - see the [TPM encryption options](../../reference/configuration/block/volumeconfig#VolumeConfig.encryption.keys..tpm) for details.
When the UKI image is generated, the UKI is measured and expected measurements are combined into TPM unlock policy and signed with the PCR signing key.
During the boot process, `systemd-stub` component of the UKI performs measurements of the UKI sections into the TPM device.
Talos Linux during the boot appends to the PCR register the measurements of the boot phases, and once the boot reaches the point of mounting the encrypted disk partition,
the expected signed policy from the UKI is matched against measured values to unlock the TPM, and TPM unseals the disk encryption key which is then used to unlock the disk partition.
## TPM PCR measurement chain
The Trusted Platform Module (TPM) maintains Platform Configuration Registers (PCRs) that record measurements of boot components:
```mermaid theme={null}
flowchart TD
BIOS[UEFI Firmware
PCR 0-6]
SB[SecureBoot State
+ Enrolled Keys
Configurable PCRs
Default: PCR 7]
STUB[systemd-stub]
LINUX[.linux section
PCR 11]
OSREL[.osrel section
PCR 11]
CMDLINE[.cmdline section
PCR 11]
INITRD[.initrd section
PCR 11]
UCODE[.ucode section
PCR 11]
SPLASH[.splash section
PCR 11]
DTB[.dtb section
PCR 11]
DTBAUTO[.dtbauto sections
PCR 11]
EFIFW[.efifw sections
PCR 11]
HWIDS[.hwids sections
PCR 11]
UNAME[.uname section
PCR 11]
SBAT[.sbat section
PCR 11]
PCRPKEY[.pcrpkey section
PCR 11]
PHASE1[enter-initrd
PCR 11]
PHASE2[leave-initrd
PCR 11]
PHASE3[enter-machined
PCR 11]
POLICY[Signed PCR Policy
PCR 11 + Configurable PCRs
Default: PCR 11 + PCR 7]
MATCH{Policy Match
at enter-machined?}
UNSEAL[Unseal Disk Key]
PHASE4[start-the-world
PCR 11]
MOUNT[Mount Encrypted
Partitions]
SERVICES[Start Services]
FAIL[Boot Failure]
BIOS --> SB
SB --> STUB
STUB --> LINUX
LINUX --> OSREL
OSREL --> CMDLINE
CMDLINE --> INITRD
INITRD --> UCODE
UCODE --> SPLASH
SPLASH --> DTB
DTB --> DTBAUTO
DTBAUTO --> EFIFW
EFIFW --> HWIDS
HWIDS --> UNAME
UNAME --> SBAT
SBAT --> PCRPKEY
PCRPKEY --> PHASE1
PHASE1 --> PHASE2
PHASE2 --> PHASE3
PHASE3 --> POLICY
POLICY --> MATCH
MATCH -->|Yes| UNSEAL
MATCH -->|No| FAIL
UNSEAL --> MOUNT
MOUNT --> PHASE4
PHASE4 --> SERVICES
style BIOS fill:#e3f2fd
style SB fill:#f1f8e9
style STUB fill:#fff8e1
style LINUX fill:#e1f5fe
style OSREL fill:#e1f5fe
style CMDLINE fill:#e1f5fe
style INITRD fill:#e1f5fe
style UCODE fill:#e1f5fe
style SPLASH fill:#e1f5fe
style DTB fill:#e1f5fe
style DTBAUTO fill:#e1f5fe
style EFIFW fill:#e1f5fe
style HWIDS fill:#e1f5fe
style UNAME fill:#e1f5fe
style SBAT fill:#e1f5fe
style PCRPKEY fill:#e1f5fe
style PHASE1 fill:#f3e5f5
style PHASE2 fill:#f3e5f5
style PHASE3 fill:#fce4ec
style PHASE4 fill:#f3e5f5
style POLICY fill:#fff3e0
style MATCH fill:#e8f5e8
style FAIL fill:#ffebee
style MOUNT fill:#e0f2f1
style SERVICES fill:#e0f2f1
```
During the upgrade, as long as the new UKI contains PCR policy signed with the same PCR signing key, and the configured PCR states have not changed, the disk partition will be unlocked successfully.
By default, disk encryption is tied to the state of **PCR 7** (SecureBoot state) in addition to **PCR 11** (boot integrity), so that it unlocks only if both the boot chain is valid and SecureBoot is enabled with the expected key set. However, **the PCR binding is fully configurable** via the `VolumeConfig` `tpm.pcrs` option - see the [TPM encryption options](../../reference/configuration/block/volumeconfig#VolumeConfig.encryption.keys..tpm) for details.
## Other boot options
Unified Kernel Image (UKI) is a UEFI-bootable image which can be booted directly from the UEFI firmware skipping the `systemd-boot` bootloader.
In network boot mode, the UKI can be used directly as well, as it contains the full set of boot assets required to boot Talos Linux.
When SecureBoot is enabled, the UKI image ignores any kernel command line arguments passed to it, but rather uses the kernel command line arguments embedded into the UKI image itself.
If kernel command line arguments need to be changed, the UKI image needs to be rebuilt with the new kernel command line arguments.
## SecureBoot with custom keys
### Generating the keys
Talos requires two set of keys to be used for the SecureBoot process:
* SecureBoot key is used to sign the boot assets and it is enrolled into the UEFI firmware.
* PCR Signing Key is used to sign the TPM policy, which is used to seal the disk encryption key.
The same key might be used for both, but it is recommended to use separate keys for each purpose.
Talos provides a utility to generate the keys, but existing PKI can be used as well:
```shell theme={null}
$ talosctl gen secureboot uki --common-name "SecureBoot Key"
writing _out/uki-signing-cert.pem
writing _out/uki-signing-cert.der
writing _out/uki-signing-key.pem
```
The generated certificate and private key are written to disk in PEM-encoded format (RSA 4096-bit key).
The certificate is also written in DER format for the systems which expect the certificate in DER format.
PCR signing key can be generated with:
```shell theme={null}
$ talosctl gen secureboot pcr
writing _out/pcr-signing-key.pem
```
The file containing the private key is written to disk in PEM-encoded format (RSA 2048-bit key).
Optionally, UEFI automatic key enrollment database can be generated using the `_out/uki-signing-*` files as input:
```shell theme={null}
$ talosctl gen secureboot database
writing _out/db.auth
writing _out/KEK.auth
writing _out/PK.auth
```
These files can be used to enroll the keys into the UEFI firmware automatically when booting from a SecureBoot ISO while UEFI firmware is in the setup mode.
> **Note** : UEFI decides what Secure Boot trusts. By default, `talosctl gen secureboot ...` generates a self-signed UKI signing certificate and `PK.auth/KEK.auth/db.auth` for enrollment. You can also generate your own version of these files which uses other signing keys and certificate authorities specific to your environment.
### Generating the SecureBoot assets
Once the keys are generated, they can be used to sign the Talos boot assets to generate required ISO images, PXE boot assets, disk images, installer containers, etc.
In this guide we will generate a SecureBoot ISO image and an installer image.
Next, the installer image should be generated to install Talos to disk on a SecureBoot-enabled system:
The generated container image should be pushed to some container registry which Talos can access during the installation, e.g.:
The generated ISO and installer images might be further customized with system extensions, extra kernel command line arguments, etc.
# Boot Assets
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/boot-assets
Creating customized Talos boot assets, disk images, ISO and installer images.
Talos Linux provides boot images via [Image Factory](#image-factory), but these images
can be customized further for a specific use case:
* adding [system extensions](../build-and-extend-talos/custom-images-and-development/system-extensions)
* updating [kernel command line arguments](../reference/kernel)
* using custom `META` contents, e.g. for [metal network configuration](../platform-specific-installations/bare-metal-platforms/metal-network-configuration)
* generating [SecureBoot](./bare-metal-platforms/secureboot) images signed with a custom key
* generating disk images for SBC's (Single Board Computers)
There are two ways to generate Talos boot assets:
* using [Image Factory](#image-factory) service (recommended)
* manually using [imager](#imager) container image (advanced)
Image Factory is easier to use, but it only produces images for official Talos Linux releases, official Talos Linux system extensions
and official Talos Overlays.
The `imager` container can be used to generate images from `main` branch, with local changes, or with custom system extensions.
## Image Factory
[Image Factory](../learn-more/image-factory) is a service that generates Talos boot assets on-demand.
Image Factory allows to generate boot assets for the official Talos Linux releases, official Talos Linux system extensions
and official Talos Overlays.
The main concept of the Image Factory is a *schematic* which defines the customization of the boot asset.
Once the schematic is configured, Image Factory can be used to pull various Talos Linux images, ISOs, installer images, PXE booting bare-metal machines across different architectures,
versions of Talos and platforms.
Sidero Labs maintains a public Image Factory instance at [https://factory.talos.dev](https://factory.talos.dev).
Image Factory provides a simple [UI](https://factory.talos.dev) to prepare schematics and retrieve asset links.
### Example: Bare-metal with Image Factory
Let's assume we want to boot Talos on a bare-metal machine with Intel CPU and add a `gvisor` container runtime to the image.
Also we want to disable predictable network interface names with `net.ifnames=0` kernel argument.
First, let's create the schematic file `bare-metal.yaml`:
```yaml theme={null}
# bare-metal.yaml
customization:
extraKernelArgs:
- net.ifnames=0
systemExtensions:
officialExtensions:
- siderolabs/gvisor
- siderolabs/intel-ucode
```
> The schematic doesn't contain system extension versions, Image Factory will pick the correct version matching Talos Linux release.
And now we can upload the schematic to the Image Factory to retrieve its ID:
```sh theme={null}
curl -X POST \
--data-binary @bare-metal.yaml \
https://factory.talos.dev/schematics
```
```json theme={null}
{"id":"b8e8fbbe1b520989e6c52c8dc8303070cb42095997e76e812fa8892393e1d176"}
```
The returned schematic ID `b8e8fbbe1b520989e6c52c8dc8303070cb42095997e76e812fa8892393e1d176` we will use to generate the boot assets.
> The schematic ID is based on the schematic contents, so uploading the same schematic will return the same ID.
Now we have two options to boot our bare-metal machine:
*
using ISO image (download it and burn to a CD/DVD or USB stick)
*
PXE booting via iPXE script
> The Image Factory URL contains both schematic ID and Talos version, and both can be changed to generate different boot assets.
Once the bare-metal machine is booted up for the first time, it will require Talos Linux `installer` image to be installed on the disk.
The `installer` image will be produced by the Image Factory as well:
Once installed, the machine can be upgraded to a new version of Talos by referencing new installer image:
```shell theme={null}
talosctl upgrade \
--image \
factory.talos.dev/metal-installer/b8e8fbbe1b520989e6c52c8dc8303070cb42095997e76e812fa8892393e1d176:
```
Same way upgrade process can be used to transition to a new set of system extensions: generate new schematic with the new set of system extensions, and upgrade the machine to the new schematic ID:
### Example: Raspberry Pi generic with Image Factory
Let's assume we want to boot Talos on a Raspberry Pi with `iscsi-tools` system extension.
First, let's create the schematic file `rpi_generic.yaml`:
```yaml theme={null}
# rpi_generic.yaml
overlay:
name: rpi_generic
image: siderolabs/sbc-raspberrypi
customization:
systemExtensions:
officialExtensions:
- siderolabs/iscsi-tools
```
> The schematic doesn't contain any system extension or overlay versions, Image Factory will pick the correct version matching Talos Linux release.
And now we can upload the schematic to the Image Factory to retrieve its ID:
```sh theme={null}
curl -X POST \
--data-binary @rpi_generic.yaml \
https://factory.talos.dev/schematics
```
```json theme={null}
{"id":"0db665edfda21c70194e7ca660955425d16cec2aa58ff031e2abf72b7c328585"}
```
The returned schematic ID `0db665edfda21c70194e7ca660955425d16cec2aa58ff031e2abf72b7c328585` we will use to generate the boot assets.
> The schematic ID is based on the schematic contents, so uploading the same schematic will return the same ID.
Now we can [download the metal arm64 image](https://factory.talos.dev/image/0db665edfda21c70194e7ca660955425d16cec2aa58ff031e2abf72b7c328585/\$\{release_v1_13}/metal-arm64.raw.xz) (download it and burn to a boot media)
> The Image Factory URL contains both schematic ID and Talos version, and both can be changed to generate different boot assets.
Once installed, the machine can be upgraded to a new version of Talos by referencing new installer image:
```shell theme={null}
talosctl upgrade \
--image \
factory.talos.dev/metal-installer/0db665edfda21c70194e7ca660955425d16cec2aa58ff031e2abf72b7c328585:
```
Same way upgrade process can be used to transition to a new set of system extensions: generate new schematic with the new set of system extensions, and upgrade the machine to the new schematic ID:
### Example: AWS with Image Factory
Talos Linux is installed on AWS from a disk image (AWS AMI), so only a single boot asset is required.
Let's assume we want to boot Talos on AWS with `gvisor` container runtime system extension.
First, let's create the schematic file `aws.yaml`:
```yaml theme={null}
# aws.yaml
customization:
systemExtensions:
officialExtensions:
- siderolabs/gvisor
```
And now we can upload the schematic to the Image Factory to retrieve its ID:
```sh theme={null}
curl -X POST \
--data-binary @aws.yaml \
https://factory.talos.dev/schematics
```
```json theme={null}
{"id":"d9ff89777e246792e7642abd3220a616afb4e49822382e4213a2e528ab826fe5"}
```
The returned schematic ID `d9ff89777e246792e7642abd3220a616afb4e49822382e4213a2e528ab826fe5` we will use to generate the boot assets.
Now we can download the AWS disk image from the Image Factory:
Now the `aws-amd64.raw.xz` file contains the customized Talos AWS disk image which can be uploaded as an AMI to [AWS](./cloud-platforms/aws).
Once the AWS VM is created from the AMI, it can be upgraded to a different Talos version or a different schematic using `talosctl upgrade`:
## Imager
A custom disk image, boot asset can be generated by using the Talos Linux `imager` container: ghcr.io/siderolabs/imager:.
The `imager` container image can be checked by [verifying its signature](../security/source-talos-images).
The generation process can be run with a simple `docker run` command:
A quick guide to the flags used for `docker run`:
* `--rm` flag removes the container after the run (as it's not going to be used anymore)
* `-t` attaches a terminal for colorized output, it can be removed if used in scripts
* `-v $PWD/_out:/secureboot:ro` mounts the SecureBoot keys into the container (can be skipped if not generating SecureBoot image)
* `-v $PWD/_out:/out` mounts the output directory (where the generated image will be placed) into the container
The `` argument to the `imager` defines the base profile to be used for the image generation.
There are several built-in profiles:
* `iso` builds a Talos ISO image (see [ISO](./bare-metal-platforms/iso))
* `secureboot-iso` builds a Talos ISO image with SecureBoot (see [SecureBoot](./bare-metal-platforms/secureboot))
* `metal` builds a generic disk image for bare-metal machines
* `secureboot-metal` builds a generic disk image for bare-metal machines with SecureBoot
* `secureboot-installer` builds an installer container image with SecureBoot (see [SecureBoot](./bare-metal-platforms/secureboot))
* `aws`, `gcp`, `azure`, etc. builds a disk image for a specific Talos platform
The base profile can be customized with the additional flags to the imager:
* `--arch` specifies the architecture of the image to be generated (default: host architecture)
* `--meta` sets initial `META` values using the format `--meta =`. If the value contains structured data, it must be properly quoted JSON (or YAML). For example: `--meta 12='{"machineLabels":{"key":"value"}}'`.
* `--extra-kernel-arg` allows to customize the kernel command line arguments.
Default kernel arg can be removed by prefixing the argument with a `-`.
For example `-console` removes all `console=` arguments, whereas `-console=tty0` removes the `console=tty0` default argument.
* `--system-extension-image` allows to install a system extension into the image
* `--image-cache` allows to use a [local image cache](../configure-your-talos-cluster/images-container-runtime/image-cache)
### Extension image reference
While Image Factory automatically resolves the extension name into a matching container image for a specific version of Talos, `imager` requires the full explicit container image reference.
The `imager` also allows to install custom extensions which are not part of the official Talos Linux system extensions.
To get the official Talos Linux system extension container image reference matching a Talos release, use the [following command](https://github.com/siderolabs/extensions?tab=readme-ov-file#installing-extensions):
This command is using [crane](https://github.com/google/go-containerregistry/blob/main/cmd/crane/README.md) tool, but any other tool which allows to export the image contents can be used.
For each Talos release, the `ghcr.io/siderolabs/extensions:VERSION` image contains a pinned reference to each system extension container image.
### Overlay image reference
While Image Factory automatically resolves the overlay name into a matching container image for a specific version of Talos, `imager` requires the full explicit container image reference.
The `imager` also allows to install custom overlays which are not part of the official Talos overlays.
To get the official Talos overlays container image reference matching a Talos release, use the following command:
This command is using [crane](https://github.com/google/go-containerregistry/blob/main/cmd/crane/README.md) tool, but any other tool which allows to export the image contents can be used.
For each Talos release, the `ghcr.io/siderolabs/overlays:VERSION` image contains a pinned reference to each overlay container image.
### Pulling from private registries
Talos Linux official images are all public, but when pulling a custom image from a private registry, the `imager` might need authentication to access the images.
The `imager` container when pulling images supports following methods to authenticate to an external registry:
* for `ghcr.io` registry, `GITHUB_TOKEN` can be provided as an environment variable;
* for other registries, `~/.docker/config.json` can be mounted into the container from the host:
* another option is to use a `DOCKER_CONFIG` environment variable, and the path will be `$DOCKER_CONFIG/config.json` in the container;
* the third option is to mount Podman's auth file at `$XDG_RUNTIME_DIR/containers/auth.json`.
### Example: Bare-metal with Imager
Let's assume we want to boot Talos on a bare-metal machine with Intel CPU and add a `gvisor` container runtime to the image.
Also we want to disable predictable network interface names with `net.ifnames=0` kernel argument and replace the Talos default `console` arguments and add a custom `console` arg.
First, let's lookup extension images for Intel CPU microcode updates and `gvisor` container runtime in the [extensions repository](https://github.com/siderolabs/extensions):
Now we can generate the ISO image with the following command:
Now the `_out/metal-amd64.iso` contains the customized Talos ISO image.
If the machine is going to be booted using PXE, we can instead generate kernel and initramfs images:
**Generate the kernel image**
**Generate the initramfs image (with extensions)**
Now the `_out/kernel-amd64` and `_out/initramfs-amd64` contain the customized Talos kernel and initramfs images.
The extra kernel args are not used now, as they are set via the PXE boot process, and can't be embedded into the kernel or initramfs.
As the next step, we should generate a custom `installer` image which contains all required system extensions (kernel args can't be specified with the installer image, but they are set in the machine configuration):
The `installer` container image should be pushed to the container registry:
Now we can use the customized `installer` image to install Talos on the bare-metal machine.
When it's time to upgrade a machine, a new `installer` image can be generated using the new version of `imager`, and updating the system extension images to the matching versions.
The custom `installer` image can now be used to upgrade Talos machine.
### Example: Raspberry Pi overlay with Imager
Let's assume we want to boot Talos on Raspberry Pi with `rpi_generic` overlay and `iscsi-tools` system extension.
First, let's lookup extension images for `iscsi-tools` in the [extensions repository](https://github.com/siderolabs/extensions):
```json theme={null}
ghcr.io/siderolabs/iscsi-tools:v0.1.4@sha256:548b2b121611424f6b1b6cfb72a1669421ffaf2f1560911c324a546c7cee655e
```
Next we'll lookup the overlay image for `rpi_generic` in the [overlays repository](https://github.com/siderolabs/overlays):
```json theme={null}
name: rpi_generic
image: ghcr.io/siderolabs/sbc-raspberrypi:v0.1.0
digest: sha256:849ace01b9af514d817b05a9c5963a35202e09a4807d12f8a3ea83657c76c863
```
Now we can generate the metal image with the following command:
Now the `_out/metal-arm64.raw.xz` is the compressed disk image which can be written to a boot media.
As the next step, we should generate a custom `installer` image which contains all required system extensions (kernel args can't be specified with the installer image, but they are set in the machine configuration):
The `installer` container image should be pushed to the container registry:
Now we can use the customized `installer` image to install Talos on Raspberry Pi.
When it's time to upgrade a machine, a new `installer` image can be generated using the new version of `imager`, and updating the system extension and overlay images to the matching versions.
The custom `installer` image can now be used to upgrade Talos machine.
### Example: AWS with Imager
Talos is installed on AWS from a disk image (AWS AMI), so only a single boot asset is required.
Let's assume we want to boot Talos on AWS with `gvisor` container runtime system extension.
First, let's lookup extension images for the `gvisor` container runtime in the [extensions repository](https://github.com/siderolabs/extensions):
Next, let's generate AWS disk image with that system extension:
Now the `_out/aws-amd64.raw.xz` contains the customized Talos AWS disk image which can be uploaded as an AMI to [AWS](./cloud-platforms/aws).
If the AWS machine is later going to be upgraded to a new version of Talos (or a new set of system extensions), generate a customized `installer` image following the steps above, and upgrade Talos to that `installer` image.
Make sure to use `--platform=aws` argument to the `imager` to generate AWS-specific installer.
### Example: Assets with system extensions from image tarballs with Imager
Some advanced features of `imager` are currently not exposed via command line arguments like `--system-extension-image`.
To access them nonetheless it is possible to supply `imager` with a `profile.yaml` instead.
Let's use these advanced features to build a bare-metal installer using a system extension from a private registry.
First use `crane` on a host with access to the private registry to export the extension image into a tarball.
```shell theme={null}
crane export \
/:latest \
```
When can then reference the tarball in a suitable `profile.yaml` for our intended architecture and output.
In this case we want to build an `amd64`, bare-metal installer.
To build the asset we pass `profile.yaml` to `imager` via stdin
### Example: Adding embedded machine configuration with Imager
Talos supports embedding a machine configuration into the boot asset. The machine configuration might either be a full machine configuration or just a subset of machine configuration
documents required e.g. to setup static network configuration.
The configuration is embedded as a virtual system extension, so it should be embedded either into the initial boot asset (e.g. ISO), or provided throughout a lifecycle via an installer image.
The example here shows how to embed a machine configuration into an ISO image.
First, let's create a machine configuration file `machine.yaml`:
```yaml theme={null}
# machine.yaml
apiVersion: v1alpha1
kind: WatchdogTimerConfig
device: /dev/watchdog0
timeout: 5m
```
This configuration file should be mounted into the `imager` container, so one of the way is to put it to the output directory `_out` which is mounted into the container:
```shell theme={null}
cp machine.yaml _out/
```
Now we can generate the ISO image which embeds this machine configuration with the following command:
Once the machine is booted from this ISO, the embedded configuration will be [automatically applied](../configure-your-talos-cluster/system-configuration/acquire) to the machine.
Use the following command to check whether the embedded configuration is present in the boot image:
```sh theme={null}
talosctl get extensions
```
```shell theme={null}
NODE NAMESPACE TYPE ID VERSION NAME VERSION
172.20.0.5 runtime ExtensionStatus 0 1 embedded-config 5484261e2ab5d34501ce9ad7875c6c0919875cd44863ee2a8a4d06bf0bd0eef7
```
The embedded configuration is immutable, so to change it, a new boot asset should be generated.
When the embedded configuration is processed the following log message is printed:
```shell theme={null}
initialized embedded config processing {"component": "controller-runtime", "controller": "config.AcquireController", "path": "/usr/local/etc/talos/config.yaml"}
```
# Akamai
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/cloud-platforms/akamai
Creating a cluster via the CLI on Akamai Cloud (Linode).
## Creating a Talos Linux cluster on Akamai connected cloud via the CLI
This guide will demonstrate how to create a highly available Kubernetes cluster with one worker using the [Akamai Connected Cloud](https://www.linode.com/) provider.
[Akamai Connected Cloud](https://www.linode.com/) has a very well-documented [REST API](https://www.linode.com/docs/api/), and an open-source [CLI](https://www.linode.com/docs/products/tools/cli/get-started/) tool to interact with the API which will be used in this guide.
Make sure to follow [installation](https://www.linode.com/docs/products/tools/cli/get-started/#installing-the-linode-cli) and authentication instructions for the `linode-cli` tool.
[jq](https://stedolan.github.io/jq/) and [talosctl](../../getting-started/talosctl#talosctl) also needs to be installed
### Upload image
Download the Akamai image `akamai-amd64.raw.gz` from
Image Factory
.
Upload the image
```bash theme={null}
export REGION=us-ord
linode-cli image-upload --region ${REGION} --label talos akamai-amd64.raw.gz
```
### Create a load balancer
```bash theme={null}
export REGION=us-ord
linode-cli nodebalancers create --region ${REGION} --no-defaults --label talos
export NODEBALANCER_ID=$(linode-cli nodebalancers list --label talos --format id --text --no-headers)
linode-cli nodebalancers config-create --port 443 --protocol tcp --check connection ${NODEBALANCER_ID}
```
### Create the machine configuration files
Using the IP address (or DNS name, if you have created one) of the load balancer, generate the base configuration files for the Talos machines.
Also note that the load balancer forwards port 443 to port 6443 on the associated nodes, so we should use 443 as the port in the config definition:
```bash theme={null}
export NODEBALANCER_IP=$(linode-cli nodebalancers list --label talos --format ipv4 --text --no-headers)
talosctl gen config talos-kubernetes-akamai https://${NODEBALANCER_IP} --with-examples=false
```
### Create the Linodes
#### Create the control plane nodes
> Although root passwords are not used by Talos, Linode requires that a root password be associated with a linode during creation.
Run the following commands to create three control plane nodes:
```bash theme={null}
export IMAGE_ID=$(linode-cli images list --label talos --format id --text --no-headers)
export NODEBALANCER_ID=$(linode-cli nodebalancers list --label talos --format id --text --no-headers)
export NODEBALANCER_CONFIG_ID=$(linode-cli nodebalancers configs-list ${NODEBALANCER_ID} --format id --text --no-headers)
export REGION=us-ord
export LINODE_TYPE=g6-standard-4
export ROOT_PW=$(pwgen 16)
for id in $(seq 3); do
linode_label="talos-control-plane-${id}"
# create linode
linode-cli linodes create \
--no-defaults \
--root_pass ${ROOT_PW} \
--type ${LINODE_TYPE} \
--region ${REGION} \
--image ${IMAGE_ID} \
--label ${linode_label} \
--private_ip true \
--tags talos-control-plane \
--group "talos-control-plane" \
--metadata.user_data "$(base64 -i ./controlplane.yaml)"
# change kernel to "direct disk"
linode_id=$(linode-cli linodes list --label ${linode_label} --format id --text --no-headers)
confiig_id=$(linode-cli linodes configs-list ${linode_id} --format id --text --no-headers)
linode-cli linodes config-update ${linode_id} ${confiig_id} --kernel "linode/direct-disk"
# add machine to nodebalancer
private_ip=$(linode-cli linodes list --label ${linode_label} --format ipv4 --json | jq -r ".[0].ipv4[1]")
linode-cli nodebalancers node-create ${NODEBALANCER_ID} ${NODEBALANCER_CONFIG_ID} --label ${linode_label} --address ${private_ip}:6443
done
```
#### Create the worker nodes
> Although root passwords are not used by Talos, Linode requires that a root password be associated with a linode during creation.
Run the following to create a worker node:
```bash theme={null}
export IMAGE_ID=$(linode-cli images list --label talos --format id --text --no-headers)
export REGION=us-ord
export LINODE_TYPE=g6-standard-4
export LINODE_LABEL="talos-worker-1"
export ROOT_PW=$(pwgen 16)
linode-cli linodes create \
--no-defaults \
--root_pass ${ROOT_PW} \
--type ${LINODE_TYPE} \
--region ${REGION} \
--image ${IMAGE_ID} \
--label ${LINODE_LABEL} \
--private_ip true \
--tags talos-worker \
--group "talos-worker" \
--metadata.user_data "$(base64 -i ./worker.yaml)"
linode_id=$(linode-cli linodes list --label ${LINODE_LABEL} --format id --text --no-headers)
config_id=$(linode-cli linodes configs-list ${linode_id} --format id --text --no-headers)
linode-cli linodes config-update ${linode_id} ${config_id} --kernel "linode/direct-disk"
```
### Bootstrap etcd
Set the `endpoints` and `nodes`:
```bash theme={null}
export LINODE_LABEL=talos-control-plane-1
export LINODE_IP=$(linode-cli linodes list --label ${LINODE_LABEL} --format ipv4 --json | jq -r ".[0].ipv4[0]")
talosctl --talosconfig talosconfig config endpoint ${LINODE_IP}
talosctl --talosconfig talosconfig config node ${LINODE_IP}
```
Bootstrap `etcd`:
```bash theme={null}
talosctl --talosconfig talosconfig bootstrap
```
### Retrieve the `kubeconfig`
At this point, we can retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl --talosconfig talosconfig kubeconfig .
```
We can also watch the cluster bootstrap via:
```bash theme={null}
talosctl --talosconfig talosconfig health
```
Alternatively, we can also watch the node overview, logs and real-time metrics dashboard via:
```bash theme={null}
talosctl --talosconfig talosconfig dashboard
```
# AWS
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/cloud-platforms/aws
Creating a cluster via the AWS CLI.
## Creating a cluster via the AWS CLI
In this guide we will create an HA Kubernetes cluster with 3 control plane nodes across 3 availability zones.
You should have an existing AWS account and have the AWS CLI installed and configured.
If you need more information on AWS specifics, please see the [official AWS documentation](https://docs.aws.amazon.com).
To install the dependencies for this tutorial you can use homebrew on macOS or Linux:
```bash theme={null}
brew install siderolabs/tap/talosctl kubectl jq curl xz
```
If you would like to create infrastructure via `terraform` or `opentofu` please see the example in the [contrib repository](https://github.com/siderolabs/contrib/tree/main/examples/terraform/aws).
> Note: this guide is not a production set up and steps were tested in `bash` and `zsh` shells.
### Create AWS resources
We will be creating a control plane with 3 Ec2 instances spread across 3 availability zones.
It is recommended to not use the default VPC so we will create a new one for this tutorial.
Change to your desired region and CIDR block and create a VPC:
> Make sure your subnet does not overlap with `10.244.0.0/16` or `10.96.0.0/12` the [default pod and services subnets in Kubernetes](../../troubleshooting/troubleshooting#conflict-on-kubernetes-and-host-subnets).
```bash theme={null}
AWS_REGION="us-west-2"
IPV4_CIDR="10.1.0.0/18"
VPC_ID=$(aws ec2 create-vpc \
--cidr-block $IPV4_CIDR \
--output text --query 'Vpc.VpcId')
```
### Create the subnets
Create 3 smaller CIDRs to use for each subnet in different availability zones.
Make sure to adjust these CIDRs if you changed the default value from the last command.
```bash theme={null}
IPV4_CIDRS=( "10.1.0.0/22" "10.1.4.0/22" "10.1.8.0/22" )
```
Next create a subnet in each availability zones.
> Note: If you're using zsh you need to run `setopt KSH_ARRAYS` to have arrays referenced properly.
```bash theme={null}
CIDR=0
declare -a SUBNETS
AZS=($(aws ec2 describe-availability-zones \
--query 'AvailabilityZones[].ZoneName' \
--filter "Name=state,Values=available" \
--output text | tr -s '\t' '\n' | head -n3))
for AZ in ${AZS[@]}; do
SUBNETS[$CIDR]=$(aws ec2 create-subnet \
--vpc-id $VPC_ID \
--availability-zone $AZ \
--cidr-block ${IPV4_CIDRS[$CIDR]} \
--query 'Subnet.SubnetId' \
--output text)
aws ec2 modify-subnet-attribute \
--subnet-id ${SUBNETS[$CIDR]} \
--private-dns-hostname-type-on-launch resource-name
echo ${SUBNETS[$CIDR]}
((CIDR++))
done
```
Create an internet gateway and attach it to the VPC:
```bash theme={null}
IGW_ID=$(aws ec2 create-internet-gateway \
--query 'InternetGateway.InternetGatewayId' \
--output text)
aws ec2 attach-internet-gateway \
--vpc-id $VPC_ID \
--internet-gateway-id $IGW_ID
ROUTE_TABLE_ID=$(aws ec2 describe-route-tables \
--filters "Name=vpc-id,Values=$VPC_ID" \
--query 'RouteTables[].RouteTableId' \
--output text)
aws ec2 create-route \
--route-table-id $ROUTE_TABLE_ID \
--destination-cidr-block 0.0.0.0/0 \
--gateway-id $IGW_ID
```
### Official AMI images
Official AMI image ID can be found in the `cloud-images.json` file attached to the [Talos release](https://github.com/siderolabs/talos/releases).
If using the official AMIs, you can skip to [Creating the Security group](#create-a-security-group)
### Create your own AMIs
> The use of the official Talos AMIs are recommended, but if you wish to build your own AMIs, follow the procedure below.
#### Create the S3 bucket
```bash theme={null}
aws s3api create-bucket \
--bucket $BUCKET \
--create-bucket-configuration LocationConstraint=$AWS_REGION \
--acl private
```
#### Create the `vmimport` role
In order to create an AMI, ensure that the `vmimport` role exists as described in the [official AWS documentation](https://docs.aws.amazon.com/vm-import/latest/userguide/required-permissions.html).
Note that the role should be associated with the S3 bucket we created above.
#### Create the image snapshot
First, download the AWS image from Image Factory:
Copy the RAW disk to S3 and import it as a snapshot:
```bash theme={null}
aws s3 cp disk.raw s3://$BUCKET/talos-aws-tutorial.raw
$SNAPSHOT_ID=$(aws ec2 import-snapshot \
--region $REGION \
--description "Talos kubernetes tutorial" \
--disk-container "Format=raw,UserBucket={S3Bucket=$BUCKET,S3Key=talos-aws-tutorial.raw}" \
--query 'SnapshotId' \
--output text)
```
To check on the status of the import, run:
```bash theme={null}
aws ec2 describe-import-snapshot-tasks \
--import-task-ids
```
Once the `SnapshotTaskDetail.Status` indicates `completed`, we can register the image.
#### Register the image
```bash theme={null}
AMI=$(aws ec2 register-image \
--block-device-mappings "DeviceName=/dev/xvda,VirtualName=talos,Ebs={DeleteOnTermination=true,SnapshotId=$SNAPSHOT_ID,VolumeSize=4,VolumeType=gp2}" \
--root-device-name /dev/xvda \
--virtualization-type hvm \
--architecture x86_64 \
--ena-support \
--name talos-aws-tutorial-ami \
--query 'ImageId' \
--output text)
```
We now have an AMI we can use to create our cluster.
### Create a security group
```bash theme={null}
SECURITY_GROUP_ID=$(aws ec2 create-security-group \
--vpc-id $VPC_ID \
--group-name talos-aws-tutorial-sg \
--description "Security Group for EC2 instances to allow ports required by Talos" \
--query 'GroupId' \
--output text)
```
Using the security group from above, allow all internal traffic within the same security group:
```bash theme={null}
aws ec2 authorize-security-group-ingress \
--group-id $SECURITY_GROUP_ID \
--protocol all \
--port 0 \
--source-group $SECURITY_GROUP_ID
```
Expose the Talos (50000) and Kubernetes API.
> Note: This is only required for the control plane nodes.
> For a production environment you would want separate private subnets for worker nodes.
```bash theme={null}
aws ec2 authorize-security-group-ingress \
--group-id $SECURITY_GROUP_ID \
--ip-permissions \
IpProtocol=tcp,FromPort=50000,ToPort=50000,IpRanges="[{CidrIp=0.0.0.0/0}]" \
IpProtocol=tcp,FromPort=6443,ToPort=6443,IpRanges="[{CidrIp=0.0.0.0/0}]" \
--query 'SecurityGroupRules[].SecurityGroupRuleId' \
--output text
```
We will bootstrap Talos with a MachineConfig via user-data it will never be exposed to the internet without certificate authentication.
We enable KubeSpan in this tutorial so you need to allow inbound UDP for the Wireguard port:
```bash theme={null}
aws ec2 authorize-security-group-ingress \
--group-id $SECURITY_GROUP_ID \
--ip-permissions \
IpProtocol=tcp,FromPort=51820,ToPort=51820,IpRanges="[{CidrIp=0.0.0.0/0}]" \
--query 'SecurityGroupRules[].SecurityGroupRuleId' \
--output text
```
### Create a load balancer
The load balancer is used for a stable Kubernetes API endpoint.
```bash theme={null}
LOAD_BALANCER_ARN=$(aws elbv2 create-load-balancer \
--name talos-aws-tutorial-lb \
--subnets $(echo ${SUBNETS[@]}) \
--type network \
--ip-address-type ipv4 \
--query 'LoadBalancers[].LoadBalancerArn' \
--output text)
LOAD_BALANCER_DNS=$(aws elbv2 describe-load-balancers \
--load-balancer-arns $LOAD_BALANCER_ARN \
--query 'LoadBalancers[].DNSName' \
--output text)
```
Now create a target group for the load balancer:
```bash theme={null}
TARGET_GROUP_ARN=$(aws elbv2 create-target-group \
--name talos-aws-tutorial-tg \
--protocol TCP \
--port 6443 \
--target-type instance \
--vpc-id $VPC_ID \
--query 'TargetGroups[].TargetGroupArn' \
--output text)
LISTENER_ARN=$(aws elbv2 create-listener \
--load-balancer-arn $LOAD_BALANCER_ARN \
--protocol TCP \
--port 6443 \
--default-actions Type=forward,TargetGroupArn=$TARGET_GROUP_ARN \
--query 'Listeners[].ListenerArn' \
--output text)
```
### Create the machine configuration files
We will create a [machine config patch](../../configure-your-talos-cluster/system-configuration/patching#rfc6902-json-patches) to use the AWS time servers.
You can create [additional patches](../../reference/configuration/v1alpha1/config) to customize the configuration as needed.
```bash theme={null}
cat < time-server-patch.yaml
machine:
time:
servers:
- 169.254.169.123
EOF
```
Using the DNS name of the loadbalancer created earlier, generate the base configuration files for the Talos machines.
```bash theme={null}
talosctl gen config talos-k8s-aws-tutorial https://${LOAD_BALANCER_DNS}:6443 \
--with-examples=false \
--with-docs=false \
--with-kubespan \
--install-disk /dev/xvda \
--config-patch '@time-server-patch.yaml'
```
> Note that the generated configs are too long for AWS userdata field if the `--with-examples` and `--with-docs` flags are not passed.
### Create the EC2 instances
> Note: There is a known issue that prevents Talos from running on T2 instance types.
> Please use T3 if you need burstable instance types.
#### Create the control plane nodes
```bash theme={null}
declare -a CP_INSTANCES
INSTANCE_INDEX=0
for SUBNET in ${SUBNETS[@]}; do
CP_INSTANCES[${INSTANCE_INDEX}]=$(aws ec2 run-instances \
--image-id $AMI \
--subnet-id $SUBNET \
--instance-type t3.small \
--user-data file://controlplane.yaml \
--associate-public-ip-address \
--security-group-ids $SECURITY_GROUP_ID \
--count 1 \
--tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=talos-aws-tutorial-cp-$INSTANCE_INDEX}]" \
--query 'Instances[].InstanceId' \
--output text)
echo ${CP_INSTANCES[${INSTANCE_INDEX}]}
((INSTANCE_INDEX++))
done
```
#### Create the worker nodes
For the worker nodes we will create a new launch template with the `worker.yaml` machine configuration and create an autoscaling group.
```bash theme={null}
WORKER_LAUNCH_TEMPLATE_ID=$(aws ec2 create-launch-template \
--launch-template-name talos-aws-tutorial-worker \
--launch-template-data '{
"ImageId":"'$AMI'",
"InstanceType":"t3.small",
"UserData":"'$(base64 -w0 worker.yaml)'",
"NetworkInterfaces":[{
"DeviceIndex":0,
"AssociatePublicIpAddress":true,
"Groups":["'$SECURITY_GROUP_ID'"],
"DeleteOnTermination":true
}],
"BlockDeviceMappings":[{
"DeviceName":"/dev/xvda",
"Ebs":{
"VolumeSize":20,
"VolumeType":"gp3",
"DeleteOnTermination":true
}
}],
"TagSpecifications":[{
"ResourceType":"instance",
"Tags":[{
"Key":"Name",
"Value":"talos-aws-tutorial-worker"
}]
}]}' \
--query 'LaunchTemplate.LaunchTemplateId' \
--output text)
```
```bash theme={null}
aws autoscaling create-auto-scaling-group \
--auto-scaling-group-name talos-aws-tutorial-worker \
--min-size 1 \
--max-size 3 \
--desired-capacity 1 \
--availability-zones $(echo ${AZS[@]}) \
--target-group-arns $TARGET_GROUP_ARN \
--launch-template "LaunchTemplateId=${WORKER_LAUNCH_TEMPLATE_ID}" \
--vpc-zone-identifier $(echo ${SUBNETS[@]} | tr ' ' ',')
```
### Configure the load balancer
Now, using the load balancer target group's ARN, and the **PrivateIpAddress** from the controlplane instances that you created :
```bash theme={null}
for INSTANCE in ${CP_INSTANCES[@]}; do
aws elbv2 register-targets \
--target-group-arn $TARGET_GROUP_ARN \
--targets Id=$(aws ec2 describe-instances \
--instance-ids $INSTANCE \
--query 'Reservations[].Instances[].InstanceId' \
--output text)
done
```
### Export the `talosconfig` file
Export the `talosconfig` file so commands sent to Talos will be authenticated.
```bash theme={null}
export TALOSCONFIG=$(pwd)/talosconfig
```
### Bootstrap `etcd`
```bash theme={null}
WORKER_INSTANCES=( $(aws autoscaling \
describe-auto-scaling-instances \
--query 'AutoScalingInstances[?AutoScalingGroupName==`talos-aws-tutorial-worker`].InstanceId' \
--output text) )
```
Set the `endpoints` (the control plane node to which `talosctl` commands are sent) and `nodes` (the nodes that the command operates on):
```bash theme={null}
talosctl config endpoints $(aws ec2 describe-instances \
--instance-ids ${CP_INSTANCES[*]} \
--query 'Reservations[].Instances[].PublicIpAddress' \
--output text)
talosctl config nodes $(aws ec2 describe-instances \
--instance-ids $(echo ${CP_INSTANCES[1]}) \
--query 'Reservations[].Instances[].PublicIpAddress' \
--output text)
```
Bootstrap `etcd`:
```bash theme={null}
talosctl bootstrap
```
You can now watch as your cluster bootstraps, by using
```bash theme={null}
talosctl health
```
This command will take a few minutes for the nodes to start etcd, reach quorum and start the Kubernetes control plane.
You can also watch the performance of a node, via:
```bash theme={null}
talosctl dashboard
```
### Retrieve the `kubeconfig`
When the cluster is healthy you can retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl kubeconfig .
export KUBECONFIG=$(pwd)/kubeconfig
```
And use standard `kubectl` commands.
```bash theme={null}
kubectl get nodes
```
## Cleanup resources
If you would like to delete all of the resources you created during this tutorial you can run the following commands.
```bash theme={null}
aws elbv2 delete-listener --listener-arn $LISTENER_ARN
aws elbv2 delete-target-group --target-group-arn $TARGET_GROUP_ARN
aws elbv2 delete-load-balancer --load-balancer-arn $LOAD_BALANCER_ARN
aws autoscaling update-auto-scaling-group \
--auto-scaling-group-name talos-aws-tutorial-worker \
--min-size 0 \
--max-size 0 \
--desired-capacity 0
aws ec2 terminate-instances --instance-ids ${CP_INSTANCES[@]} ${WORKER_INSTANCES[@]} \
--query 'TerminatingInstances[].InstanceId' \
--output text
aws autoscaling delete-auto-scaling-group \
--auto-scaling-group-name talos-aws-tutorial-worker \
--force-delete
aws ec2 delete-launch-template --launch-template-id $WORKER_LAUNCH_TEMPLATE_ID
while $(aws ec2 describe-instances \
--instance-ids ${CP_INSTANCES[@]} ${WORKER_INSTANCES[@]} \
--query 'Reservations[].Instances[].[InstanceId,State.Name]' \
--output text | grep -q shutting-down); do \
echo "waiting for instances to terminate"; sleep 5s
done
aws ec2 detach-internet-gateway --vpc-id $VPC_ID --internet-gateway-id $IGW_ID
aws ec2 delete-internet-gateway --internet-gateway-id $IGW_ID
aws ec2 delete-security-group --group-id $SECURITY_GROUP_ID
for SUBNET in ${SUBNETS[@]}; do
aws ec2 delete-subnet --subnet-id $SUBNET
done
aws ec2 delete-vpc --vpc-id $VPC_ID
rm -f controlplane.yaml worker.yaml talosconfig kubeconfig time-server-patch.yaml disk.raw
```
# Azure
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/cloud-platforms/azure
Creating a cluster via the CLI on Azure.
## Creating a cluster via the CLI
In this guide we will create an HA Kubernetes cluster with 1 worker node.
We assume existing [Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/), and some familiarity with Azure.
If you need more information on Azure specifics, please see the [official Azure documentation](https://docs.microsoft.com/en-us/azure/).
### Environment setup
We'll make use of the following environment variables throughout the setup.
Edit the variables below with your correct information.
```bash theme={null}
# Storage account to use
export STORAGE_ACCOUNT="StorageAccountName"
# Storage container to upload to
export STORAGE_CONTAINER="StorageContainerName"
# Resource group name
export GROUP="ResourceGroupName"
# Location
export LOCATION="centralus"
# Get storage account connection string based on info above
export CONNECTION=$(az storage account show-connection-string \
-n $STORAGE_ACCOUNT \
-g $GROUP \
-o tsv)
```
### Create the image
First, download the Azure image from [Image Factory](https://factory.talos.dev/).
Once downloaded, untar with `tar -xvf /path/to/azure-amd64.tar.gz`
#### Upload the VHD
Once you have pulled down the image, you can upload it to blob storage with:
```bash theme={null}
az storage blob upload \
--connection-string $CONNECTION \
--container-name $STORAGE_CONTAINER \
-f /path/to/extracted/talos-azure.vhd \
-n talos-azure.vhd
```
#### Register the image
Now that the image is present in our blob storage, we'll register it.
```bash theme={null}
az image create \
--name talos \
--source https://$STORAGE_ACCOUNT.blob.core.windows.net/$STORAGE_CONTAINER/talos-azure.vhd \
--os-type linux \
-g $GROUP
```
### Network infrastructure
#### Virtual networks and security groups
Once the image is prepared, we'll want to work through setting up the network.
Issue the following to create a network security group and add rules to it.
```bash theme={null}
# Create vnet
az network vnet create \
--resource-group $GROUP \
--location $LOCATION \
--name talos-vnet \
--subnet-name talos-subnet
# Create network security group
az network nsg create -g $GROUP -n talos-sg
# Client -> apid
az network nsg rule create \
-g $GROUP \
--nsg-name talos-sg \
-n apid \
--priority 1001 \
--destination-port-ranges 50000 \
--direction inbound
# Trustd
az network nsg rule create \
-g $GROUP \
--nsg-name talos-sg \
-n trustd \
--priority 1002 \
--destination-port-ranges 50001 \
--direction inbound
# etcd
az network nsg rule create \
-g $GROUP \
--nsg-name talos-sg \
-n etcd \
--priority 1003 \
--destination-port-ranges 2379-2380 \
--direction inbound
# Kubernetes API Server
az network nsg rule create \
-g $GROUP \
--nsg-name talos-sg \
-n kube \
--priority 1004 \
--destination-port-ranges 6443 \
--direction inbound
```
#### Load balancer
We will create a public ip, load balancer, and a health check that we will use for our control plane.
```bash theme={null}
# Create public ip
az network public-ip create \
--resource-group $GROUP \
--name talos-public-ip \
--allocation-method static
# Create lb
az network lb create \
--resource-group $GROUP \
--name talos-lb \
--public-ip-address talos-public-ip \
--frontend-ip-name talos-fe \
--backend-pool-name talos-be-pool
# Create health check
az network lb probe create \
--resource-group $GROUP \
--lb-name talos-lb \
--name talos-lb-health \
--protocol tcp \
--port 6443
# Create lb rule for 6443
az network lb rule create \
--resource-group $GROUP \
--lb-name talos-lb \
--name talos-6443 \
--protocol tcp \
--frontend-ip-name talos-fe \
--frontend-port 6443 \
--backend-pool-name talos-be-pool \
--backend-port 6443 \
--probe-name talos-lb-health
```
#### Network interfaces
In Azure, we have to pre-create the NICs for our control plane so that they can be associated with our load balancer.
```bash theme={null}
for i in $( seq 0 1 2 ); do
# Create public IP for each nic
az network public-ip create \
--resource-group $GROUP \
--name talos-controlplane-public-ip-$i \
--allocation-method static
# Create nic
az network nic create \
--resource-group $GROUP \
--name talos-controlplane-nic-$i \
--vnet-name talos-vnet \
--subnet talos-subnet \
--network-security-group talos-sg \
--public-ip-address talos-controlplane-public-ip-$i\
--lb-name talos-lb \
--lb-address-pools talos-be-pool
done
# NOTES:
# Talos can detect PublicIPs automatically if PublicIP SKU is Basic.
# Use `--sku Basic` to set SKU to Basic.
```
### Cluster configuration
With our networking bits setup, we'll fetch the IP for our load balancer and create our configuration files.
```bash theme={null}
LB_PUBLIC_IP=$(az network public-ip show \
--resource-group $GROUP \
--name talos-public-ip \
--query "ipAddress" \
--output tsv)
talosctl gen config talos-k8s-azure-tutorial https://${LB_PUBLIC_IP}:6443
```
### Compute creation
We are now ready to create our azure nodes.
Azure allows you to pass Talos machine configuration to the virtual machine at bootstrap time via
`user-data` or `custom-data` methods.
Talos supports only `custom-data` method, machine configuration is available to the VM only on the first boot.
#### Manual image upload
```bash theme={null}
# Create availability set
az vm availability-set create \
--name talos-controlplane-av-set \
-g $GROUP
# Create the controlplane nodes
for i in $( seq 0 1 2 ); do
az vm create \
--name talos-controlplane-$i \
--image talos \
--custom-data ./controlplane.yaml \
-g $GROUP \
--admin-username talos \
--generate-ssh-keys \
--verbose \
--boot-diagnostics-storage $STORAGE_ACCOUNT \
--os-disk-size-gb 20 \
--nics talos-controlplane-nic-$i \
--availability-set talos-controlplane-av-set \
--no-wait
done
# Create worker node
az vm create \
--name talos-worker-0 \
--image talos \
--vnet-name talos-vnet \
--subnet talos-subnet \
--custom-data ./worker.yaml \
-g $GROUP \
--admin-username talos \
--generate-ssh-keys \
--verbose \
--boot-diagnostics-storage $STORAGE_ACCOUNT \
--nsg talos-sg \
--os-disk-size-gb 20 \
--no-wait
# NOTES:
# `--admin-username` and `--generate-ssh-keys` are required by the az cli,
# but are not actually used by talos
# `--os-disk-size-gb` is the backing disk for Kubernetes and any workload containers
# `--boot-diagnostics-storage` is to enable console output which may be necessary
# for troubleshooting
```
### Bootstrap etcd
You should now be able to interact with your cluster with `talosctl`.
We will need to discover the public IP for our first control plane node first.
```bash theme={null}
CONTROL_PLANE_0_IP=$(az network public-ip show \
--resource-group $GROUP \
--name talos-controlplane-public-ip-0 \
--query "ipAddress" \
--output tsv)
```
Set the `endpoints` and `nodes`:
```bash theme={null}
talosctl --talosconfig talosconfig config endpoint $CONTROL_PLANE_0_IP
talosctl --talosconfig talosconfig config node $CONTROL_PLANE_0_IP
```
Bootstrap `etcd`:
```bash theme={null}
talosctl --talosconfig talosconfig bootstrap
```
### Retrieve the `kubeconfig`
At this point we can retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl --talosconfig talosconfig kubeconfig .
```
# CloudStack
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/cloud-platforms/cloudstack
Creating a cluster via the CLI (cmk) on Apache CloudStack.
## Creating a Talos Linux Cluster on Apache CloudStack via the CMK CLI
In this guide we will create an single node Kubernetes cluster in Apache CloudStack.
We assume Apache CloudStack is already running in a basic configuration - and some familiarity with Apache CloudStack.
We will be using the [CloudStack Cloudmonkey](https://github.com/apache/cloudstack-cloudmonkey) CLI tool.
Please see the [official Apache CloudStack documentation](https://docs.cloudstack.apache.org/en/latest/) for information related to Apache CloudStack.
### Obtain the Talos image
Download the Talos CloudStack image `cloudstack-amd64.raw.gz` from the [Image Factory](https://factory.talos.dev).
> Note: the minimum version of Talos required to support Apache CloudStack is v1.8.0.
Using an upload method of your choice, upload the image to a Apache CloudStack.
You might be able to use the "Register Template from URL" to download the image directly from the Image Factory.
> Note: CloudStack does not seem to like compressed images, so you might have to download the image to a local webserver, uncompress it and let CloudStack fetch the image from there instead.
> Alternatively, you can try to remove `.gz` from URL to fetch an uncompressed image from the Image Factory.
### Get required variables
Next we will get a number of required variables and export them for later use:
#### Get image template ID
```bash theme={null}
$ cmk list templates templatefilter=self | jq -r '.template[] | [.id, .name] | @tsv' | sort -k2
01813d29-1253-4080-8d29-d405d94148af Talos 1.8.0
...
$ export IMAGE_ID=01813d29-1253-4080-8d29-d405d94148af
```
#### Get Zone ID
Get a list of Zones and select the relevant zone
```bash theme={null}
$ cmk list zones | jq -r '.zone[] | [.id, .name] | @tsv' | sort -k2
a8c71a6f-2e09-41ed-8754-2d4dd8783920 fsn1
9d38497b-d810-42ab-a772-e596994d21d2 fsn2
...
$ export ZONE_ID=a8c71a6f-2e09-41ed-8754-2d4dd8783920
```
#### Get service offering ID
Get a list of service offerings (instance types) and select the desired offering
```bash theme={null}
$ cmk list serviceofferings | jq -r '.serviceoffering[] | [.id, .memory, .cpunumber, .name] | @tsv' | sort -k4
82ac8c87-22ee-4ec3-8003-c80b09efe02c 2048 2 K8S-CP-S
c7f5253e-e1f1-4e33-a45e-eb2ebbc65fd4 4096 2 K8S-WRK-S
...
$ export SERVICEOFFERING_ID=82ac8c87-22ee-4ec3-8003-c80b09efe02c
```
#### Get network ID
Get a list of networks and select the relevant network for your cluster.
```bash theme={null}
$ cmk list networks zoneid=${ZONE_ID} | jq -r '.network[] | [.id, .type, .name] | @tsv' | sort -k3
f706984f-9dd1-4cb8-9493-3fba1f0de7e3 Isolate demo
143ed8f1-3cc5-4ba2-8717-457ad993cf25 Isolated talos
...
$ export NETWORK_ID=143ed8f1-3cc5-4ba2-8717-457ad993cf25
```
#### Get next free public IP address and ID
To create a loadbalancer for the K8S API Endpoint, find the next available public IP address in the zone.
(In this test environment, the 10.0.0.0/24 RFC-1918 IP range has been configured as "Public IP addresses")
```bash theme={null}
$ cmk list publicipaddresses zoneid=${ZONE_ID} state=free forvirtualnetwork=true | jq -r '.publicipaddress[] | [.id, .ipaddress] | @tsv' | sort -k2
1901d946-3797-48aa-a113-8fb730b0770a 10.0.0.102
fa207d0e-c8f8-4f09-80f0-d45a6aac77eb 10.0.0.103
aa397291-f5dc-4903-b299-277161b406cb 10.0.0.104
...
$ export PUBLIC_IPADDRESS=10.0.0.102
$ export PUBLIC_IPADDRESS_ID=1901d946-3797-48aa-a113-8fb730b0770a
```
#### Acquire and associate Public IP Address
Acquire and associate the public IP address with the network we selected earlier.
```bash theme={null}
$ cmk associateIpAddress ipaddress=${PUBLIC_IPADDRESS} networkid=${NETWORK_ID}
{
"ipaddress": {
...,
"ipaddress": "10.0.0.102",
...
}
}
```
#### Create LB and FW rule using the public IP address
Create a Loadbalancer for the K8S API Endpoint.
> Note: The "create loadbalancerrule" also takes care of creating a corresponding firewallrule.
```bash theme={null}
$ cmk create loadbalancerrule algorithm=roundrobin name="k8s-api" privateport=6443 publicport=6443 openfirewall=true publicipid=${PUBLIC_IPADDRESS_ID} cidrlist=0.0.0.0/0
{
"loadbalancer": {
...
"name": "k8s-api",
"networkid": "143ed8f1-3cc5-4ba2-8717-457ad993cf25",
"privateport": "6443",
"publicip": "10.0.0.102",
"publicipid": "1901d946-3797-48aa-a113-8fb730b0770a",
"publicport": "6443",
...
}
}
```
### Create the Talos configuration files
Finally it's time to generate the Talos configuration files, using the Public IP address assigned to the loadbalancer.
```bash theme={null}
$ talosctl gen config talos-cloudstack https://${PUBLIC_IPADDRESS}:6443 --with-docs=false --with-examples=false
created controlplane.yaml
created worker.yaml
created talosconfig
```
Make any adjustments to the `controlplane.yaml` and/or `worker.yaml` as you like.
> Note: Remember to validate!
#### Create Talos VM
Next we will create the actual VM and supply the `controlplane.yaml` as base64 encoded `userdata`.
```bash theme={null}
$ cmk deploy virtualmachine zoneid=${ZONE_ID} templateid=${IMAGE_ID} serviceofferingid=${SERVICEOFFERING_ID} networkIds=${NETWORK_ID} name=talosdemo usersdata=$(base64 controlplane.yaml | tr -d '\n')
{
"virtualmachine": {
"account": "admin",
"affinitygroup": [],
"cpunumber": 2,
"cpuspeed": 2000,
"cpuused": "0.3%",
...
}
}
```
#### Get Talos VM ID and internal IP address
Get the ID of our newly created VM.
(Also available in the full output of the above command.)
```bash theme={null}
$ cmk list virtualmachines | jq -r '.virtualmachine[] | [.id, .ipaddress, .name]|@tsv' | sort -k3
9c119627-cb38-4b64-876b-ca2b79820b5a 10.1.1.154 srv03
545099fc-ec2d-4f32-915d-b0c821cfb634 10.1.1.97 srv04
d37aeca4-7d1f-45cd-9a4d-97fdbf535aa1 10.1.1.243 talosdemo
$ export VM_ID=d37aeca4-7d1f-45cd-9a4d-97fdbf535aa1
$ export VM_IP=10.1.1.243
```
#### Get load balancer ID
Obtain the ID of the `loadbalancerrule` we created earlier.
```bash theme={null}
$ cmk list loadbalancerrules | jq -r '.loadbalancerrule[]| [.id, .publicip, .name] | @tsv' | sort -k2
ede6b711-b6bc-4ade-9e48-4b3f5aa59934 10.0.0.102 k8s-api
1bad3c46-96fa-4f50-a4fc-9a46a54bc350 10.0.0.197 ac0b5d98cf6a24d55a4fb2f9e240c473-tcp-443
$ export LB_RULE_ID=ede6b711-b6bc-4ade-9e48-4b3f5aa59934
```
#### Assign Talos VM to load balancer
With the ID of the VM and the load balancer, we can assign the VM to the `loadbalancerrule`, making the K8S API endpoint available via the Load Balancer
```bash theme={null}
cmk assigntoloadbalancerrule id=${LB_RULE_ID} virtualmachineids=${VM_ID}
```
### Bootstrap etcd
Once the Talos VM has booted, it time to bootstrap etcd.
Configure `talosctl` with IP addresses of the control plane node's IP address.
Set the `endpoints` and `nodes`:
```bash theme={null}
talosctl --talosconfig talosconfig config endpoint ${VM_IP}
talosctl --talosconfig talosconfig config node ${VM_IP}
```
Next, bootstrap `etcd`:
```bash theme={null}
talosctl --talosconfig talosconfig bootstrap
```
### Retrieve the `kubeconfig`
At this point we can retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl --talosconfig talosconfig kubeconfig .
```
We can also watch the cluster bootstrap via:
```bash theme={null}
talosctl --talosconfig talosconfig dashboard
```
# DigitalOcean
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/cloud-platforms/digitalocean
Creating a cluster via the CLI on DigitalOcean.
## Creating a Talos Linux cluster on Digital Ocean via the CLI
In this guide we will create an HA Kubernetes cluster with 1 worker node, in the NYC region.
We assume an existing [Space](https://www.digitalocean.com/docs/spaces/), and some familiarity with DigitalOcean.
If you need more information on DigitalOcean specifics, please see the [official DigitalOcean documentation](https://www.digitalocean.com/docs/).
### Create the image
Download the DigitalOcean image `digital-ocean-amd64.raw.gz` from the
Image Factory
.
> Note: the minimum version of Talos required to support Digital Ocean is v1.3.3.
Using an upload method of your choice (`doctl` does not have Spaces support), upload the image to a space.
(It's easy to drag the image file to the space using DigitalOcean's web console.)
*Note:* Make sure you upload the file as `public`.
Now, create an image using the URL of the uploaded image:
```bash theme={null}
export REGION=nyc3
doctl compute image create \
--region $REGION \
--image-description talos-digital-ocean-tutorial \
--image-url https://$SPACENAME.$REGION.digitaloceanspaces.com/digital-ocean-amd64.raw.gz \
Talos
```
Save the image ID.
We will need it when creating droplets.
### Create a load balancer
```bash theme={null}
doctl compute load-balancer create \
--region $REGION \
--name talos-digital-ocean-tutorial-lb \
--tag-name talos-digital-ocean-tutorial-control-plane \
--health-check protocol:tcp,port:6443,check_interval_seconds:10,response_timeout_seconds:5,healthy_threshold:5,unhealthy_threshold:3 \
--forwarding-rules entry_protocol:tcp,entry_port:443,target_protocol:tcp,target_port:6443
```
Note the returned ID of the load balancer.
We will need the IP of the load balancer.
Using the ID of the load balancer, run:
```bash theme={null}
doctl compute load-balancer get --format IP
```
Note that it may take a few minutes before the load balancer is provisioned, so repeat this command until it returns with the IP address.
### Create the machine configuration files
Using the IP address (or DNS name, if you have created one) of the loadbalancer, generate the base configuration files for the Talos machines.
Also note that the load balancer forwards port 443 to port 6443 on the associated nodes, so we should use 443 as the port in the config definition:
```bash theme={null}
$ talosctl gen config talos-k8s-digital-ocean-tutorial https://:443
created controlplane.yaml
created worker.yaml
created talosconfig
```
### Create the Droplets
#### Create a dummy SSH key
> Although SSH is not used by Talos, DigitalOcean requires that an SSH key be associated with a droplet during creation.
> We will create a dummy key that can be used to satisfy this requirement.
```bash theme={null}
doctl compute ssh-key create --public-key "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDbl0I1s/yOETIKjFr7mDLp8LmJn6OIZ68ILjVCkoN6lzKmvZEqEm1YYeWoI0xgb80hQ1fKkl0usW6MkSqwrijoUENhGFd6L16WFL53va4aeJjj2pxrjOr3uBFm/4ATvIfFTNVs+VUzFZ0eGzTgu1yXydX8lZMWnT4JpsMraHD3/qPP+pgyNuI51LjOCG0gVCzjl8NoGaQuKnl8KqbSCARIpETg1mMw+tuYgaKcbqYCMbxggaEKA0ixJ2MpFC/kwm3PcksTGqVBzp3+iE5AlRe1tnbr6GhgT839KLhOB03j7lFl1K9j1bMTOEj5Io8z7xo/XeF2ZQKHFWygAJiAhmKJ dummy@dummy.local" dummy
```
Note the ssh key ID that is returned - we will use it in creating the droplets.
#### Create the control plane nodes
Run the following commands to create three control plane nodes:
```bash theme={null}
doctl compute droplet create \
--region $REGION \
--image \
--size s-2vcpu-4gb \
--enable-private-networking \
--tag-names talos-digital-ocean-tutorial-control-plane \
--user-data-file controlplane.yaml \
--ssh-keys \
talos-control-plane-1
doctl compute droplet create \
--region $REGION \
--image \
--size s-2vcpu-4gb \
--enable-private-networking \
--tag-names talos-digital-ocean-tutorial-control-plane \
--user-data-file controlplane.yaml \
--ssh-keys \
talos-control-plane-2
doctl compute droplet create \
--region $REGION \
--image \
--size s-2vcpu-4gb \
--enable-private-networking \
--tag-names talos-digital-ocean-tutorial-control-plane \
--user-data-file controlplane.yaml \
--ssh-keys \
talos-control-plane-3
```
Note the droplet ID returned for the first control plane node.
#### Create the worker nodes
Run the following to create a worker node:
```bash theme={null}
doctl compute droplet create \
--region $REGION \
--image \
--size s-2vcpu-4gb \
--enable-private-networking \
--user-data-file worker.yaml \
--ssh-keys \
talos-worker-1
```
### Bootstrap etcd
To configure `talosctl` we will need the first control plane node's IP:
```bash theme={null}
doctl compute droplet get --format PublicIPv4
```
Set the `endpoints` and `nodes`:
```bash theme={null}
talosctl --talosconfig talosconfig config endpoint
talosctl --talosconfig talosconfig config node
```
Bootstrap `etcd`:
```bash theme={null}
talosctl --talosconfig talosconfig bootstrap
```
### Retrieve the `kubeconfig`
At this point we can retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl --talosconfig talosconfig kubeconfig .
```
We can also watch the cluster bootstrap via:
```bash theme={null}
talosctl --talosconfig talosconfig health
```
# Exoscale
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/cloud-platforms/exoscale
Creating a cluster via the CLI using exoscale.com
Talos is known to work on exoscale.com; however, it is currently undocumented.
# GCP
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/cloud-platforms/gcp
Creating a cluster via the CLI on Google Cloud Platform.
## Creating a cluster via the CLI
In this guide, we will create an HA Kubernetes cluster in GCP with 1 worker node.
We will assume an existing [Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets), and some familiarity with Google Cloud.
If you need more information on Google Cloud specifics, please see the [official Google documentation](https://cloud.google.com/docs/).
[jq](https://stedolan.github.io/jq/) and [talosctl](../../getting-started/quickstart#talosctl) also needs to be installed
## Manual setup
### Environment setup
We'll make use of the following environment variables throughout the setup.
Edit the variables below with your correct information.
```bash theme={null}
# Storage account to use
export STORAGE_BUCKET="StorageBucketName"
# Region
export REGION="us-central1"
```
### Create the image
First, download the Google Cloud image from [Image Factory](https://factory.talos.dev/).
These images are called `gcp-$ARCH.tar.gz`.
#### Upload the image
Once you have downloaded the image, you can upload it to your storage bucket with:
```bash theme={null}
gsutil cp /path/to/gcp-amd64.tar.gz gs://$STORAGE_BUCKET
```
#### Register the image
Now that the image is present in our bucket, we'll register it.
```bash theme={null}
gcloud compute images create talos \
--source-uri=gs://$STORAGE_BUCKET/gcp-amd64.tar.gz \
--guest-os-features=VIRTIO_SCSI_MULTIQUEUE
```
### Network infrastructure
#### Load balancers and firewalls
Once the image is prepared, we'll want to work through setting up the network.
Issue the following to create a firewall, load balancer, and their required components.
`130.211.0.0/22` and `35.191.0.0/16` are the GCP [Load Balancer IP ranges](https://cloud.google.com/load-balancing/docs/health-checks#fw-rule)
```bash theme={null}
# Create Instance Group
gcloud compute instance-groups unmanaged create talos-ig \
--zone $REGION-b
# Create port for IG
gcloud compute instance-groups set-named-ports talos-ig \
--named-ports tcp6443:6443 \
--zone $REGION-b
# Create health check
gcloud compute health-checks create tcp talos-health-check --port 6443
# Create backend
gcloud compute backend-services create talos-be \
--global \
--protocol TCP \
--health-checks talos-health-check \
--timeout 5m \
--port-name tcp6443
# Add instance group to backend
gcloud compute backend-services add-backend talos-be \
--global \
--instance-group talos-ig \
--instance-group-zone $REGION-b
# Create tcp proxy
gcloud compute target-tcp-proxies create talos-tcp-proxy \
--backend-service talos-be \
--proxy-header NONE
# Create LB IP
gcloud compute addresses create talos-lb-ip --global
# Forward 443 from LB IP to tcp proxy
gcloud compute forwarding-rules create talos-fwd-rule \
--global \
--ports 443 \
--address talos-lb-ip \
--target-tcp-proxy talos-tcp-proxy
# Create firewall rule for health checks
gcloud compute firewall-rules create talos-controlplane-firewall \
--source-ranges 130.211.0.0/22,35.191.0.0/16 \
--target-tags talos-controlplane \
--allow tcp:6443
# Create firewall rule to allow talosctl access
gcloud compute firewall-rules create talos-controlplane-talosctl \
--source-ranges 0.0.0.0/0 \
--target-tags talos-controlplane \
--allow tcp:50000
```
### Cluster configuration
With our networking bits setup, we'll fetch the IP for our load balancer and create our configuration files.
```bash theme={null}
LB_PUBLIC_IP=$(gcloud compute forwarding-rules describe talos-fwd-rule \
--global \
--format json \
| jq -r .IPAddress)
talosctl gen config talos-k8s-gcp-tutorial https://${LB_PUBLIC_IP}:443
```
Additionally, you can specify [machine configuration patches](../../configure-your-talos-cluster/system-configuration/patching#configuration-patching-with-talosctl-cli) which will be applied during the config generation.
### Compute creation
We are now ready to create our GCP nodes.
```bash theme={null}
# Create the control plane nodes.
for i in $( seq 0 2 ); do
gcloud compute instances create talos-controlplane-$i \
--image talos \
--zone $REGION-b \
--tags talos-controlplane,talos-controlplane-$i \
--boot-disk-size 20GB \
--metadata-from-file=user-data=./controlplane.yaml
done
# Add control plane nodes to instance group
for i in $( seq 0 2 ); do
gcloud compute instance-groups unmanaged add-instances talos-ig \
--zone $REGION-b \
--instances talos-controlplane-$i
done
# Create worker
gcloud compute instances create talos-worker-0 \
--image talos \
--zone $REGION-b \
--boot-disk-size 20GB \
--metadata-from-file=user-data=./worker.yaml \
--tags talos-worker-$i
```
### Bootstrap etcd
You should now be able to interact with your cluster with `talosctl`.
We will need to discover the public IP for our first control plane node first.
```bash theme={null}
CONTROL_PLANE_0_IP=$(gcloud compute instances describe talos-controlplane-0 \
--zone $REGION-b \
--format json \
| jq -r '.networkInterfaces[0].accessConfigs[0].natIP')
```
Set the `endpoints` and `nodes`:
```bash theme={null}
talosctl --talosconfig talosconfig config endpoint $CONTROL_PLANE_0_IP
talosctl --talosconfig talosconfig config node $CONTROL_PLANE_0_IP
```
Bootstrap `etcd`:
```bash theme={null}
talosctl --talosconfig talosconfig bootstrap
```
### Retrieve the `kubeconfig`
At this point we can retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl --talosconfig talosconfig kubeconfig .
```
### Cleanup
```bash theme={null}
# cleanup VM's
gcloud compute instances delete \
talos-worker-0 \
talos-controlplane-0 \
talos-controlplane-1 \
talos-controlplane-2
# cleanup firewall rules
gcloud compute firewall-rules delete \
talos-controlplane-talosctl \
talos-controlplane-firewall
# cleanup forwarding rules
gcloud compute forwarding-rules delete \
talos-fwd-rule
# cleanup addresses
gcloud compute addresses delete \
talos-lb-ip
# cleanup proxies
gcloud compute target-tcp-proxies delete \
talos-tcp-proxy
# cleanup backend services
gcloud compute backend-services delete \
talos-be
# cleanup health checks
gcloud compute health-checks delete \
talos-health-check
# cleanup unmanaged instance groups
gcloud compute instance-groups unmanaged delete \
talos-ig
# cleanup Talos image
gcloud compute images delete \
talos
```
## Using GCP deployment manager
Using GCP deployment manager automatically creates a Google Storage bucket and uploads the Talos image to it.
Once the deployment is complete the generated `talosconfig` and `kubeconfig` files are uploaded to the bucket.
By default this setup creates a three node control plane and a single worker in `us-west1-b`
First we need to create a folder to store our deployment manifests and perform all subsequent operations from that folder.
```bash theme={null}
mkdir -p talos-gcp-deployment
cd talos-gcp-deployment
```
### Getting the deployment manifests
We need to download two deployment manifests for the deployment from the Talos github repository.
### Updating the config
Now we need to update the local `config.yaml` file with any required changes such as changing the default zone, Talos version, machine sizes, nodes count etc.
An example `config.yaml` file is shown below:
#### Enabling external cloud provider
Note: The `externalCloudProvider` property is set to `false` by default.
The manifest used for deploying the ccm (cloud controller manager) is currently using the GCP ccm provided by openshift since there are no public images for the [ccm](https://github.com/kubernetes/cloud-provider-gcp) yet.
> Since the routes controller is disabled while deploying the CCM, the CNI pods needs to be restarted after the CCM deployment is complete to remove the `node.kubernetes.io/network-unavailable` taint.
> See [Nodes network-unavailable taint not removed after installing ccm](https://github.com/kubernetes/cloud-provider-gcp/issues/291) for more information
Use a custom built image for the ccm deployment if required.
### Creating the deployment
Now we are ready to create the deployment.
Confirm with `y` for any prompts.
Run the following command to create the deployment:
```bash theme={null}
# use a unique name for the deployment, resources are prefixed with the deployment name
export DEPLOYMENT_NAME=""
gcloud deployment-manager deployments create "${DEPLOYMENT_NAME}" --config config.yaml
```
### Retrieving the outputs
First we need to get the deployment outputs.
```bash theme={null}
# first get the outputs
OUTPUTS=$(gcloud deployment-manager deployments describe "${DEPLOYMENT_NAME}" --format json | jq '.outputs[]')
BUCKET_NAME=$(jq -r '. | select(.name == "bucketName").finalValue' <<< "${OUTPUTS}")
# used when cloud controller is enabled
SERVICE_ACCOUNT=$(jq -r '. | select(.name == "serviceAccount").finalValue' <<< "${OUTPUTS}")
PROJECT=$(jq -r '. | select(.name == "project").finalValue' <<< "${OUTPUTS}")
```
Note: If cloud controller manager is enabled, the below command needs to be run to allow the controller custom role to access cloud resources
```bash theme={null}
gcloud projects add-iam-policy-binding \
"${PROJECT}" \
--member "serviceAccount:${SERVICE_ACCOUNT}" \
--role roles/iam.serviceAccountUser
gcloud projects add-iam-policy-binding \
"${PROJECT}" \
--member serviceAccount:"${SERVICE_ACCOUNT}" \
--role roles/compute.admin
gcloud projects add-iam-policy-binding \
"${PROJECT}" \
--member serviceAccount:"${SERVICE_ACCOUNT}" \
--role roles/compute.loadBalancerAdmin
```
### Downloading talos and kubeconfig
In addition to the `talosconfig` and `kubeconfig` files, the storage bucket contains the `controlplane.yaml` and `worker.yaml` files used to join additional nodes to the cluster.
```bash theme={null}
gsutil cp "gs://${BUCKET_NAME}/generated/talosconfig" .
gsutil cp "gs://${BUCKET_NAME}/generated/kubeconfig" .
```
### Deploying the cloud controller manager
```bash theme={null}
kubectl \
--kubeconfig kubeconfig \
--namespace kube-system \
apply \
--filename gcp-ccm.yaml
# wait for the ccm to be up
kubectl \
--kubeconfig kubeconfig \
--namespace kube-system \
rollout status \
daemonset cloud-controller-manager
```
If the cloud controller manager is enabled, we need to restart the CNI pods to remove the `node.kubernetes.io/network-unavailable` taint.
```bash theme={null}
# restart the CNI pods, in this case flannel
kubectl \
--kubeconfig kubeconfig \
--namespace kube-system \
rollout restart \
daemonset kube-flannel
# wait for the pods to be restarted
kubectl \
--kubeconfig kubeconfig \
--namespace kube-system \
rollout status \
daemonset kube-flannel
```
### Check cluster status
```bash theme={null}
kubectl \
--kubeconfig kubeconfig \
get nodes
```
### Cleanup deployment
Warning: This will delete the deployment and all resources associated with it.
Run below if cloud controller manager is enabled
```bash theme={null}
gcloud projects remove-iam-policy-binding \
"${PROJECT}" \
--member "serviceAccount:${SERVICE_ACCOUNT}" \
--role roles/iam.serviceAccountUser
gcloud projects remove-iam-policy-binding \
"${PROJECT}" \
--member serviceAccount:"${SERVICE_ACCOUNT}" \
--role roles/compute.admin
gcloud projects remove-iam-policy-binding \
"${PROJECT}" \
--member serviceAccount:"${SERVICE_ACCOUNT}" \
--role roles/compute.loadBalancerAdmin
```
Now we can finally remove the deployment
```bash theme={null}
# delete the objects in the bucket first
gsutil -m rm -r "gs://${BUCKET_NAME}"
gcloud deployment-manager deployments delete "${DEPLOYMENT_NAME}" --quiet
```
# Hetzner
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/cloud-platforms/hetzner
Creating a cluster via the CLI (hcloud) on Hetzner.
## Upload image
Hetzner Cloud provides Talos as a Public ISO with the schematic id `ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515` (Hetzner + qemu-guest-agent) since 2025-04-23.
Minor updates of the ISO will be provided by Hetzner Cloud on a best effort basis.
If you need an ISO with a different schematic id, please email the support team to get a Talos ISO uploaded by following [issues:3599](https://github.com/siderolabs/talos/issues/3599#issuecomment-841172018) or you can prepare the image snapshot by yourself.
There are three options to upload your own:
1. Run an instance in rescue mode and replace the system OS with the Talos image
2. Use [Hashicorp Packer](https://developer.hashicorp.com/packer/integrations/hetznercloud/hcloud) to prepare an image
3. Use special utility [hcloud-upload-image](https://github.com/apricote/hcloud-upload-image/)
### Rescue mode
Create a new Server in the Hetzner console.
Enable the Hetzner Rescue System for this server and reboot.
Upon a reboot, the server will boot a special minimal Linux distribution designed for repair and reinstall.
Once running, login to the server using `ssh` to prepare the system disk by doing the following:
To make sure disk content is consistent, it is recommended to shut the server down before taking an image (snapshot).
Once shut down, simply create an image (snapshot) from the console.
You can now use this snapshot to run Talos on the cloud.
### Packer
Install [packer](https://learn.hashicorp.com/tutorials/packer/get-started-install-cli) to the local machine.
Create a config file for packer to use:
Additionally, you could create a file containing
```hcl theme={null}
arch = "arm64"
server_type = "cax11"
server_location = "fsn1"
```
and build the snapshot for arm64.
Create a new image by issuing the commands shown below.
Note that to create a new API token for your Project, switch into the Hetzner Cloud Console, choose a Project, go to Access → Security, and create a new token.
```bash theme={null}
# First you need set the API Token
export HCLOUD_TOKEN=${TOKEN}
# Upload image
packer init .
packer build .
# Save the image ID
export IMAGE_ID=
```
After doing this, you can find the snapshot in the console interface.
### hcloud-upload-image
Install `hcloud-upload-image` as described [in its repository](https://github.com/apricote/hcloud-upload-image/?tab=readme-ov-file#getting-started). You can either download the binary, build it from source or use the Docker container.
For process simplification you can use this `bash` script:
After these actions, you can find the snapshot in the console interface.
## Creating a cluster via the CLI
This section assumes you have the [hcloud console utility](https://community.hetzner.com/tutorials/howto-hcloud-cli) on your local machine.
```bash theme={null}
# Set hcloud context and api key
hcloud context create talos-tutorial
```
### Create a load balancer
Create a load balancer by issuing the commands shown below.
Save the IP/DNS name, as this info will be used in the next step.
```bash theme={null}
hcloud load-balancer create --name controlplane --network-zone eu-central --type lb11 --label 'type=controlplane'
### Result is like:
# LoadBalancer 484487 created
# IPv4: 49.12.X.X
# IPv6: 2a01:4f8:X:X::1
hcloud load-balancer add-service controlplane \
--listen-port 6443 --destination-port 6443 --protocol tcp
hcloud load-balancer add-target controlplane \
--label-selector 'type=controlplane'
```
### Create the machine configuration files
#### Generating base configurations
Using the IP/DNS name of the load balancer created earlier, generate the base configuration files for the Talos machines by issuing:
```bash theme={null}
talosctl gen config talos-k8s-hcloud-tutorial https://:6443 \
--with-examples=false --with-docs=false
```
Generating the config without examples and docs is necessary because otherwise you can easily exceed the 32 kb limit on uploadable userdata (see [issue 8805](https://github.com/siderolabs/talos/issues/8805)).
At this point, you can modify the generated configs to your liking.
Optionally, you can specify [machine configuration patches](../../configure-your-talos-cluster/system-configuration/patching#configuration-patching-with-talosctl-cli) which will be applied during the config generation.
#### Validate the configuration files
Validate any edited machine configs with:
```bash theme={null}
talosctl validate --config controlplane.yaml --mode cloud
talosctl validate --config worker.yaml --mode cloud
```
### Create the servers
We can now create our servers.
Note that you can find `IMAGE_ID` in the snapshot section of the console: `https://console.hetzner.cloud/projects/$PROJECT_ID/servers/snapshots`.
#### Create the control plane nodes
Create the control plane nodes with:
```bash theme={null}
export IMAGE_ID=
hcloud server create --name talos-control-plane-1 \
--image ${IMAGE_ID} \
--type cx22 --location hel1 \
--label 'type=controlplane' \
--user-data-from-file controlplane.yaml
hcloud server create --name talos-control-plane-2 \
--image ${IMAGE_ID} \
--type cx22 --location fsn1 \
--label 'type=controlplane' \
--user-data-from-file controlplane.yaml
hcloud server create --name talos-control-plane-3 \
--image ${IMAGE_ID} \
--type cx22 --location nbg1 \
--label 'type=controlplane' \
--user-data-from-file controlplane.yaml
```
#### Create the worker nodes
Create the worker nodes with the following command, repeating (and incrementing the name counter) as many times as desired.
```bash theme={null}
hcloud server create --name talos-worker-1 \
--image ${IMAGE_ID} \
--type cx22 --location hel1 \
--label 'type=worker' \
--user-data-from-file worker.yaml
```
### Bootstrap etcd
To configure `talosctl` we will need the first control plane node's IP.
This can be found by issuing:
```bash theme={null}
hcloud server list | grep talos-control-plane
```
Set the `endpoints` and `nodes` for your talosconfig with:
```bash theme={null}
talosctl --talosconfig talosconfig config endpoint
talosctl --talosconfig talosconfig config node
```
Bootstrap `etcd` on the first control plane node with:
```bash theme={null}
talosctl --talosconfig talosconfig bootstrap
```
After a successful bootstrap, you should see that all the members have joined:
```bash theme={null}
talosctl --talosconfig talosconfig -n get members
```
### Retrieve the `kubeconfig`
At this point we can retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl --talosconfig talosconfig kubeconfig .
```
### Install Hetzner's Cloud Controller Manager
First of all, we need to patch the Talos machine configuration used by each node:
Then run the following command:
```bash theme={null}
talosctl --talosconfig talosconfig patch machineconfig --patch-file patch.yaml --nodes
```
With that in place, we can now follow the [official instructions](https://github.com/hetznercloud/hcloud-cloud-controller-manager), ignoring the `kubeadm` related steps.
# Kubernetes
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/cloud-platforms/kubernetes
Running Talos Linux as a pod in Kubernetes.
Talos Linux can be run as a pod in Kubernetes similar to running Talos in [Docker](../local-platforms/docker).
This can be used e.g. to run controlplane nodes inside an existing Kubernetes cluster.
Talos Linux running in Kubernetes is not full Talos Linux experience, as it is running in a container using the host's kernel and network stack.
Some operations like upgrades and reboots are not supported.
## Prerequisites
* a running Kubernetes cluster
* a `talos` container image:
## Machine configuration
Machine configuration can be generated using [Getting Started](../../getting-started/getting-started) guide.
Machine install disk will ge ignored, as the install image.
The Talos version will be driven by the container image being used.
The required machine configuration patch to enable using container runtime DNS:
```yaml theme={null}
machine:
features:
hostDNS:
enabled: true
forwardKubeDNSToHost: true
```
Talos and Kubernetes API can be exposed using Kubernetes services or load balancers, so they can be accessed from outside the cluster.
## Running Talos pods
There might be many ways to run Talos in Kubernetes (StatefulSet, Deployment, single Pod), so we will only provide some basic guidance here.
### Container settings
### Submitting initial machine configuration
Initial machine configuration can be submitted using `talosctl apply-config --insecure` when the pod is running, or it can be submitted
via an environment variable `USERDATA` with base64-encoded machine configuration.
### Volume mounts
Three ephemeral mounts are required for `/run`, `/system`, and `/tmp` directories:
```yaml theme={null}
volumeMounts:
- mountPath: /run
name: run
- mountPath: /system
name: system
- mountPath: /tmp
name: tmp
```
```yaml theme={null}
volumes:
- emptyDir: {}
name: run
- emptyDir: {}
name: system
- emptyDir: {}
name: tmp
```
Several other mountpoints are required, and they should persist across pod restarts, so one should use `PersistentVolume` for them:
```yaml theme={null}
volumeMounts:
- mountPath: /system/state
name: system-state
- mountPath: /var
name: var
- mountPath: /etc/cni
name: etc-cni
- mountPath: /etc/kubernetes
name: etc-kubernetes
- mountPath: /usr/libexec/kubernetes
name: usr-libexec-kubernetes
```
# Nocloud
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/cloud-platforms/nocloud
Configuring Talos networking via the `nocloud` specification.
Talos supports [nocloud](https://cloudinit.readthedocs.io/en/latest/reference/datasources/nocloud.html) data source implementation.
On bare-metal, Talos Linux was tested to correctly parse `nocloud` configuration from the following providers:
* [phoenixNAP Bare Metal Cloud](https://phoenixnap.com/)
* [servers.com](https://www.servers.com/)
There are two ways to configure Talos server with `nocloud` platform:
* via SMBIOS "serial number" option
* using CDROM or USB-flash filesystem
> Note: This requires the nocloud image which can be downloaded from the [Image Factory](https://factory.talos.dev/).
### SMBIOS serial number
This method requires the network connection to be up (e.g. via DHCP).
Configuration is delivered from the HTTP server.
```text theme={null}
ds=nocloud-net;s=http://10.10.0.1/configs/;h=HOSTNAME
```
After the network initialization is complete, Talos fetches:
* the machine config from `http://10.10.0.1/configs/user-data`
* the network config (if available) from `http://10.10.0.1/configs/network-config`
#### SMBIOS: QEMU
Add the following flag to `qemu` command line when starting a VM:
```bash theme={null}
qemu-system-x86_64 \
...\
-smbios type=1,serial=ds=nocloud-net;s=http://10.10.0.1/configs/
```
#### SMBIOS: Proxmox
Set the source machine config through the serial number on Proxmox GUI.
![]()
You can read the VM config from a `root` shell with the command `qm config $ID` (`$ID` - VM ID number of virtual machine), you will see something like:
```shellsession theme={null}
# qm config $ID
...
smbios1: uuid=5b0f7dcf-cfe3-4bf3-87a2-1cad29bd51f9,serial=ZHM9bm9jbG91ZC1uZXQ7cz1odHRwOi8vMTAuMTAuMC4xL2NvbmZpZ3Mv,base64=1
...
```
Where serial holds the base64-encoded string version of `ds=nocloud-net;s=http://10.10.0.1/configs/`.
The serial can also be set from a `root` shell on the Proxmox server:
```shellsession theme={null}
# qm set $VM --smbios1 "uuid=5b0f7dcf-cfe3-4bf3-87a2-1cad29bd51f9,serial=$(printf '%s' 'ds=nocloud-net;s=http://10.10.0.1/configs/' | base64),base64=1"
update VM 105: -smbios1 uuid=5b0f7dcf-cfe3-4bf3-87a2-1cad29bd51f9,serial=ZHM9bm9jbG91ZC1uZXQ7cz1odHRwOi8vMTAuMTAuMC4xL2NvbmZpZ3Mv,base64=1
```
Keep in mind that if you set the serial from the command line, you must encode it as base64, and you must include the UUID and any other settings that are already set for the `smbios1` option or they will be removed.
### CDROM/USB
Talos can also get machine config from local attached storage without any prior network connection being established.
You can provide configs to the server via files on a VFAT or ISO9660 filesystem.
The filesystem volume label must be `cidata` or `CIDATA`.
#### Example: QEMU
Create and prepare Talos machine config:
```bash theme={null}
export CONTROL_PLANE_IP=192.168.1.10
talosctl gen config talos-nocloud https://$CONTROL_PLANE_IP:6443 --output-dir _out
```
Prepare cloud-init configs:
```bash theme={null}
mkdir -p iso
mv _out/controlplane.yaml iso/user-data
echo "local-hostname: controlplane-1" > iso/meta-data
cat > iso/network-config << EOF
version: 1
config:
- type: physical
name: eth0
mac_address: "52:54:00:12:34:00"
subnets:
- type: static
address: 192.168.1.10
netmask: 255.255.255.0
gateway: 192.168.1.254
EOF
```
Create cloud-init iso image
```bash theme={null}
cd iso && genisoimage -output cidata.iso -V cidata -r -J user-data meta-data network-config
```
Start the VM
```bash theme={null}
qemu-system-x86_64 \
...
-cdrom iso/cidata.iso \
...
```
#### Example: Proxmox
Proxmox can create cloud-init disk [for you](https://pve.proxmox.com/wiki/Cloud-Init_Support).
Edit the cloud-init config information in Proxmox as follows, substitute your own information as necessary:
![]()
and then add a `cicustom` param to the virtual machine's configuration from a `root` shell:
```shellsession theme={null}
# qm set 100 --cicustom user=local:snippets/controlplane-1.yml
update VM 100: -cicustom user=local:snippets/controlplane-1.yml
```
> Note: `snippets/controlplane-1.yml` is Talos machine config.
> It is usually located at `/var/lib/vz/snippets/controlplane-1.yml`.
> This file must be placed to this path manually, as Proxmox does not support snippet uploading via API/GUI.
Click on `Regenerate Image` button after the above changes are made.
# OpenStack
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/cloud-platforms/openstack
Creating a cluster via the CLI on OpenStack.
## Creating a cluster via the CLI
In this guide, we will create an HA Kubernetes cluster in OpenStack with 1 worker node.
We will assume an existing some familiarity with OpenStack.
If you need more information on OpenStack specifics, please see the [official OpenStack documentation](https://docs.openstack.org).
### Environment setup
You should have an existing openrc file.
This file will provide environment variables necessary to talk to your OpenStack cloud.
See [here](https://docs.openstack.org/newton/user-guide/common/cli-set-environment-variables-using-openstack-rc.html) for instructions on fetching this file.
### Create the image
First, download the OpenStack image from [Image Factory](https://factory.talos.dev/).
These images are called `openstack-$ARCH.tar.gz`.
Untar this file with `tar -xvf openstack-$ARCH.tar.gz`.
The resulting file will be called `disk.raw`.
#### Upload the image
Once you have the image, you can upload to OpenStack with:
```bash theme={null}
openstack image create --public --disk-format raw --file disk.raw talos
```
### Network infrastructure
#### Load balancer and network ports
Once the image is prepared, you will need to work through setting up the network.
Issue the following to create a load balancer, the necessary network ports for each control plane node, and associations between the two.
Creating loadbalancer:
```bash theme={null}
# Create load balancer, updating vip-subnet-id if necessary
openstack loadbalancer create --name talos-control-plane --vip-subnet-id public
# Create listener
openstack loadbalancer listener create --name talos-control-plane-listener --protocol TCP --protocol-port 6443 talos-control-plane
# Pool and health monitoring
openstack loadbalancer pool create --name talos-control-plane-pool --lb-algorithm ROUND_ROBIN --listener talos-control-plane-listener --protocol TCP
openstack loadbalancer healthmonitor create --delay 5 --max-retries 4 --timeout 10 --type TCP talos-control-plane-pool
```
Creating ports:
```bash theme={null}
# Create ports for control plane nodes, updating network name if necessary
openstack port create --network shared talos-control-plane-1
openstack port create --network shared talos-control-plane-2
openstack port create --network shared talos-control-plane-3
# Create floating IPs for the ports, so that you will have talosctl connectivity to each control plane
openstack floating ip create --port talos-control-plane-1 public
openstack floating ip create --port talos-control-plane-2 public
openstack floating ip create --port talos-control-plane-3 public
```
> Note: Take notice of the private and public IPs associated with each of these ports, as they will be used in the next step.
> Additionally, take node of the port ID, as it will be used in server creation.
Associate port's private IPs to loadbalancer:
```bash theme={null}
# Create members for each port IP, updating subnet-id and address as necessary.
openstack loadbalancer member create --subnet-id shared-subnet --address --protocol-port 6443 talos-control-plane-pool
openstack loadbalancer member create --subnet-id shared-subnet --address --protocol-port 6443 talos-control-plane-pool
openstack loadbalancer member create --subnet-id shared-subnet --address --protocol-port 6443 talos-control-plane-pool
```
#### Security groups
This example uses the default security group in OpenStack.
Ports have been opened to ensure that connectivity from both inside and outside the group is possible.
You will want to allow, at a minimum, ports 6443 (Kubernetes API server) and 50000 (Talos API) from external sources.
It is also recommended to allow communication over all ports from within the subnet.
### Cluster configuration
With our networking bits setup, we'll fetch the IP for our load balancer and create our configuration files.
```bash theme={null}
LB_PUBLIC_IP=$(openstack loadbalancer show talos-control-plane -f json | jq -r .vip_address)
talosctl gen config talos-k8s-openstack-tutorial https://${LB_PUBLIC_IP}:6443
```
Additionally, you can specify [machine configuration patches](../../configure-your-talos-cluster/system-configuration/patching#configuration-patching-with-talosctl-cli) which will be applied during the config generation.
### Compute creation
We are now ready to create our OpenStack nodes.
Create control plane:
```bash theme={null}
# Create control planes 2 and 3, substituting the same info.
for i in $( seq 1 3 ); do
openstack server create talos-control-plane-$i --flavor m1.small --nic port-id=talos-control-plane-$i --image talos --user-data /path/to/controlplane.yaml
done
```
Create worker:
```bash theme={null}
# Update network name as necessary.
openstack server create talos-worker-1 --flavor m1.small --network shared --image talos --user-data /path/to/worker.yaml
```
> Note: This step can be repeated to add more workers.
### Bootstrap etcd
You should now be able to interact with your cluster with `talosctl`.
We will use one of the floating IPs we allocated earlier.
It does not matter which one.
Set the `endpoints` and `nodes`:
```bash theme={null}
talosctl --talosconfig talosconfig config endpoint
talosctl --talosconfig talosconfig config node
```
Bootstrap `etcd`:
```bash theme={null}
talosctl --talosconfig talosconfig bootstrap
```
### Retrieve the `kubeconfig`
At this point we can retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl --talosconfig talosconfig kubeconfig .
```
# Oracle
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/cloud-platforms/oracle
Creating a cluster via the CLI (oci) on OracleCloud.com.
## Upload image
Oracle Cloud at the moment does not have a Talos official image.
So you can use [Bring Your Own Image (BYOI)](https://docs.oracle.com/en-us/iaas/Content/Compute/References/bringyourownimage.htm) approach.
Prepare an image for upload:
1. Generate an image using [Image Factory](https://factory.talos.dev).
2. Download the disk image artifact in qcow2 format (e.g:
Oracle arm64
)
3. Define the image metadata file called `image_metadata.json`.
Example for an `arm64` deployment:
```json theme={null}
{
"version": 2,
"externalLaunchOptions": {
"firmware": "UEFI_64",
"networkType": "PARAVIRTUALIZED",
"bootVolumeType": "PARAVIRTUALIZED",
"remoteDataVolumeType": "PARAVIRTUALIZED",
"localDataVolumeType": "PARAVIRTUALIZED",
"launchOptionsSource": "PARAVIRTUALIZED",
"pvAttachmentVersion": 2,
"pvEncryptionInTransitEnabled": true,
"consistentVolumeNamingEnabled": true
},
"imageCapabilityData": null,
"imageCapsFormatVersion": null,
"operatingSystem": "Talos",
"operatingSystemVersion": "1.7.6",
"additionalMetadata": {
"shapeCompatibilities": [
{
"internalShapeName": "VM.Standard.A1.Flex",
"ocpuConstraints": null,
"memoryConstraints": null
}
]
}
}
```
4. Create an archive containing the image and metadata called `talos-oracle-arm64.oci`:
```shell theme={null}
tar zcf oracle-arm64.oci oracle-arm64.qcow2 image_metadata.json
```
5. Upload the image to a storage bucket.
6. Create an image, using the *new* URL format for the storage bucket object.
> Note: file names depends on configuration of deployment such as architecture, adjust accordingly.
## Talos config
OracleCloud has highly available NTP service, it can be enabled in Talos machine config with:
```yaml theme={null}
machine:
time:
servers:
- 169.254.169.254
```
## Creating a cluster via the CLI
Login to the [console](https://www.oracle.com/cloud/).
And open the Cloud Shell.
### Create a network
```bash theme={null}
export cidr_block=10.0.0.0/16
export subnet_block=10.0.0.0/24
export compartment_id= # https://docs.cloud.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/cmdref/network/vcn/create.html#cmdoption-compartment-id
export vcn_id=$(oci network vcn create --cidr-block $cidr_block --display-name talos-example --compartment-id $compartment_id --query data.id --raw-output)
export rt_id=$(oci network subnet create --cidr-block $subnet_block --display-name kubernetes --compartment-id $compartment_id --vcn-id $vcn_id --query data.route-table-id --raw-output)
export ig_id=$(oci network internet-gateway create --compartment-id $compartment_id --is-enabled true --vcn-id $vcn_id --query data.id --raw-output)
oci network route-table update --rt-id $rt_id --route-rules "[{\"cidrBlock\":\"0.0.0.0/0\",\"networkEntityId\":\"$ig_id\"}]" --force
# disable firewall
export sl_id=$(oci network vcn list --compartment-id $compartment_id --query 'data[0]."default-security-list-id"' --raw-output)
oci network security-list update --security-list-id $sl_id --egress-security-rules '[{"destination": "0.0.0.0/0", "protocol": "all", "isStateless": false}]' --ingress-security-rules '[{"source": "0.0.0.0/0", "protocol": "all", "isStateless": false}]' --force
```
### Create a load balancer
Create a load balancer by issuing the commands shown below.
Save the IP/DNS name, as this info will be used in the next step.
```bash theme={null}
export subnet_id=$(oci network subnet list --compartment-id=$compartment_id --display-name kubernetes --query data[0].id --raw-output)
export network_load_balancer_id=$(oci nlb network-load-balancer create --compartment-id $compartment_id --display-name controlplane-lb --subnet-id $subnet_id --is-preserve-source-destination false --is-private false --query data.id --raw-output)
cat < talos-health-checker.json
{
"intervalInMillis": 10000,
"port": 50000,
"protocol": "TCP"
}
EOF
oci nlb backend-set create --health-checker file://talos-health-checker.json --name talos --network-load-balancer-id $network_load_balancer_id --policy TWO_TUPLE --is-preserve-source false
oci nlb listener create --default-backend-set-name talos --name talos --network-load-balancer-id $network_load_balancer_id --port 50000 --protocol TCP
cat < controlplane-health-checker.json
{
"intervalInMillis": 10000,
"port": 6443,
"protocol": "HTTPS",
"returnCode": 401,
"urlPath": "/readyz"
}
EOF
oci nlb backend-set create --health-checker file://controlplane-health-checker.json --name controlplane --network-load-balancer-id $network_load_balancer_id --policy TWO_TUPLE --is-preserve-source false
oci nlb listener create --default-backend-set-name controlplane --name controlplane --network-load-balancer-id $network_load_balancer_id --port 6443 --protocol TCP
# Save the external IP
oci nlb network-load-balancer list --compartment-id $compartment_id --display-name controlplane-lb --query 'data.items[0]."ip-addresses"'
```
### Create the machine configuration files
#### Generating base configurations
Using the IP/DNS name of the loadbalancer created earlier, generate the base configuration files for the Talos machines by issuing:
```bash theme={null}
$ talosctl gen config talos-k8s-oracle-tutorial https://:6443 --additional-sans
created controlplane.yaml
created worker.yaml
created talosconfig
```
At this point, you can modify the generated configs to your liking.
Optionally, you can specify [machine configuration patches](../../configure-your-talos-cluster/system-configuration/patching#configuration-patching-with-talosctl-cli) which will be applied during the config generation.
#### Validate the configuration files
Validate any edited machine configs with:
```bash theme={null}
$ talosctl validate --config controlplane.yaml --mode cloud
controlplane.yaml is valid for cloud mode
$ talosctl validate --config worker.yaml --mode cloud
worker.yaml is valid for cloud mode
```
### Create the servers
#### Create the control plane nodes
Create the control plane nodes with:
```bash theme={null}
export shape='VM.Standard.A1.Flex'
export subnet_id=$(oci network subnet list --compartment-id=$compartment_id --display-name kubernetes --query data[0].id --raw-output)
export image_id=$(oci compute image list --compartment-id $compartment_id --shape $shape --operating-system Talos --limit 1 --query data[0].id --raw-output)
export availability_domain=$(oci iam availability-domain list --compartment-id=$compartment_id --query data[0].name --raw-output)
export network_load_balancer_id=$(oci nlb network-load-balancer list --compartment-id $compartment_id --display-name controlplane-lb --query 'data.items[0].id' --raw-output)
cat < shape.json
{
"memoryInGBs": 4,
"ocpus": 1
}
EOF
export instance_id=$(oci compute instance launch --shape $shape --shape-config file://shape.json --availability-domain $availability_domain --compartment-id $compartment_id --image-id $image_id --subnet-id $subnet_id --display-name controlplane-1 --private-ip 10.0.0.11 --assign-public-ip true --launch-options '{"networkType":"PARAVIRTUALIZED"}' --user-data-file controlplane.yaml --query 'data.id' --raw-output)
oci nlb backend create --backend-set-name talos --network-load-balancer-id $network_load_balancer_id --port 50000 --target-id $instance_id
oci nlb backend create --backend-set-name controlplane --network-load-balancer-id $network_load_balancer_id --port 6443 --target-id $instance_id
export instance_id=$(oci compute instance launch --shape $shape --shape-config file://shape.json --availability-domain $availability_domain --compartment-id $compartment_id --image-id $image_id --subnet-id $subnet_id --display-name controlplane-2 --private-ip 10.0.0.12 --assign-public-ip true --launch-options '{"networkType":"PARAVIRTUALIZED"}' --user-data-file controlplane.yaml --query 'data.id' --raw-output)
oci nlb backend create --backend-set-name talos --network-load-balancer-id $network_load_balancer_id --port 50000 --target-id $instance_id
oci nlb backend create --backend-set-name controlplane --network-load-balancer-id $network_load_balancer_id --port 6443 --target-id $instance_id
export instance_id=$(oci compute instance launch --shape $shape --shape-config file://shape.json --availability-domain $availability_domain --compartment-id $compartment_id --image-id $image_id --subnet-id $subnet_id --display-name controlplane-3 --private-ip 10.0.0.13 --assign-public-ip true --launch-options '{"networkType":"PARAVIRTUALIZED"}' --user-data-file controlplane.yaml --query 'data.id' --raw-output)
oci nlb backend create --backend-set-name talos --network-load-balancer-id $network_load_balancer_id --port 50000 --target-id $instance_id
oci nlb backend create --backend-set-name controlplane --network-load-balancer-id $network_load_balancer_id --port 6443 --target-id $instance_id
```
#### Create the worker nodes
Create the worker nodes with the following command, repeating (and incrementing the name counter) as many times as desired.
```bash theme={null}
export subnet_id=$(oci network subnet list --compartment-id=$compartment_id --display-name kubernetes --query data[0].id --raw-output)
export image_id=$(oci compute image list --compartment-id $compartment_id --operating-system Talos --limit 1 --query data[0].id --raw-output)
export availability_domain=$(oci iam availability-domain list --compartment-id=$compartment_id --query data[0].name --raw-output)
export shape='VM.Standard.E2.1.Micro'
oci compute instance launch --shape $shape --availability-domain $availability_domain --compartment-id $compartment_id --image-id $image_id --subnet-id $subnet_id --display-name worker-1 --assign-public-ip true --user-data-file worker.yaml
oci compute instance launch --shape $shape --availability-domain $availability_domain --compartment-id $compartment_id --image-id $image_id --subnet-id $subnet_id --display-name worker-2 --assign-public-ip true --user-data-file worker.yaml
oci compute instance launch --shape $shape --availability-domain $availability_domain --compartment-id $compartment_id --image-id $image_id --subnet-id $subnet_id --display-name worker-3 --assign-public-ip true --user-data-file worker.yaml
```
### Bootstrap etcd
To configure `talosctl` we will need the first control plane node's IP.
This can be found by issuing:
```bash theme={null}
export instance_id=$(oci compute instance list --compartment-id $compartment_id --display-name controlplane-1 --query 'data[0].id' --raw-output)
oci compute instance list-vnics --instance-id $instance_id --query 'data[0]."private-ip"' --raw-output
```
Set the `endpoints` and `nodes` for your talosconfig with:
```bash theme={null}
talosctl --talosconfig talosconfig config endpoint
talosctl --talosconfig talosconfig config node
```
Bootstrap `etcd` on the first control plane node with:
```bash theme={null}
talosctl --talosconfig talosconfig bootstrap
```
### Retrieve the `kubeconfig`
At this point we can retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl --talosconfig talosconfig kubeconfig .
```
# OVHCloud
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/cloud-platforms/ovhcloud
Creating a cluster via the OpenStack CLI on OVHCloud.
In this guide, we will create an HA Kubernetes cluster in OVHCloud with 1 worker node.
We will assume an existing some familiarity with OpenStack.
If you need more information on OpenStack specifics, please see the [official OVHCloud documentation](https://docs.ovh.com/).
## Environment setup
You should have an existing openrc file.
This file will provide environment variables necessary to talk to your OVHCloud.
See [here](https://help.ovhcloud.com/csm/en-gb-public-cloud-compute-openstack-tokens?id=kb_article_view\&sysparm_article=KB0057950) for instructions on fetching this file.
Set environment variables:
```bash theme={null}
export PRIVATE_NETWORK_NAME=private-network-001
export SUBNET_NAME=subnet-001
export SUBNET_CIDR=10.0.0.0/16
export GATEWAY_NAME=gateway-001
export COMPUTE_FLAVOR_NAME=b3-8
export IMAGE_NAME=talos
export LOADBALANCER_NAME=loadbalancer-001
export LOADBALANCER_FLAVOR_NAME=small
```
## Step 1. Create the image
First, download the OpenStack image from [Image Factory](https://factory.talos.dev/).
These images are called `openstack-$ARCH.raw.xz`.
Decompress this file with `unxz openstack-$ARCH.raw.xz`.
The resulting file will be called `openstack-$ARCH.raw`.
#### Upload the image
Once you have the image, you can upload to OpenStack with:
```bash theme={null}
openstack image create $IMAGE_NAME \
--disk-format raw \
--file openstack-amd64.raw \
--progress
```
## Step 2. Network infrastructure (Optional)
If you have an existing network infrastructure, you can skip this step.
If not, you can follow the instructions below to create the necessary network infrastructure for your cluster.
We will create a private network, subnet, and gateway for our cluster.
This will allow our cluster to communicate with the outside world and with each other.
### Private network, subnet, gateway
```bash theme={null}
# Create private network
openstack network create \
--provider-network-type vrack \
--provider-segment 1 \
--mtu 1500 \
--enable-port-security \
--internal \
$PRIVATE_NETWORK_NAME
export PRIVATE_NETWORK_ID=$(openstack network show $PRIVATE_NETWORK_NAME -f value -c id)
# Create subnet
openstack subnet create \
--network $PRIVATE_NETWORK_ID \
--subnet-range $SUBNET_CIDR \
--dns-nameserver 1.1.1.1 \
--dns-nameserver 8.8.8.8 \
$SUBNET_NAME
# Create Gateway
openstack router create $GATEWAY_NAME --external-gateway Ext-Net
# Add the subnet to the router
openstack router add subnet \
$GATEWAY_NAME \
$SUBNET_NAME
```
### Security groups
This example uses the default security group in OpenStack.
Ports have been opened to ensure that connectivity from both inside and outside the group is possible.
You will want to allow, at a minimum, ports 6443 (Kubernetes API server) and 50000 (Talos API) from external sources.
It is also recommended to allow communication over all ports from within the subnet.
## Step 3. Compute
Once the image is uploaded and the network infrastructure is in place, we can create our compute instances.
We will create 3 control plane nodes.
```bash theme={null}
# Create control planes 2 and 3, substituting the same info.
for i in $( seq 1 3 ); do
openstack server create talos-control-plane-$i --flavor $COMPUTE_FLAVOR_NAME --nic net-id=$PRIVATE_NETWORK_ID --image $IMAGE_NAME
done
```
We have now created our compute instances, but we still need to add floating IPs to allow for communication with the control plane nodes.
```bash theme={null}
# Create floating IPs and associate them with the control plane nodes
for i in $( seq 1 3 ); do
openstack floating ip create --port $(openstack port list --server talos-control-plane-$i -f value -c ID) Ext-Net
done
```
### Load balancer
Once the compute instances are created, we need to create a load balancer to load balance traffic to the control plane nodes.
Running the following commands can take a few minutes to complete as the load balancer is being provisioned.
```bash theme={null}
# Create a Load Balancer
export SUBNET_ID=$(openstack subnet show $SUBNET_NAME -f value -c id)
openstack loadbalancer create --name $LOADBALANCER_NAME --vip-subnet-id $SUBNET_ID --flavor $LOADBALANCER_FLAVOR_NAME --wait
openstack floating ip create --port $(openstack loadbalancer show $LOADBALANCER_NAME -f value -c VIP_PORT_ID) Ext-Net
# Create a listener for the load balancer
openstack loadbalancer listener create --name talos-control-plane-listener --protocol TCP --protocol-port 6443 $LOADBALANCER_NAME --wait
# Create a pool and health monitor for the load balancer
openstack loadbalancer pool create --name talos-control-plane-pool --lb-algorithm ROUND_ROBIN --listener talos-control-plane-listener --protocol TCP --wait
openstack loadbalancer healthmonitor create --delay 5 --max-retries 4 --timeout 10 --type TCP talos-control-plane-pool
# Add members to the load balancer pool, substituting the private IPs of the control plane nodes and the protocol port (6443)
for i in $( seq 1 3 ); do
openstack loadbalancer member create --subnet-id $SUBNET_ID --address $(openstack floating ip list --port $(openstack port list --server talos-control-plane-$i -f value -c ID) -f value -c "Floating IP Address") --protocol-port 6443 talos-control-plane-pool --wait
done
```
## Step 4. Cluster configuration
Now that we have our compute instances and load balancer set up, we can generate our cluster configuration files with `talosctl`.
We need to get the Public IP of the Load Balancer as this is our Kubernetes API endpoint, and the Public IPs of the control plane nodes as addional-sans for the API certificate.
```bash theme={null}
# Get the Public IP of the Load Balancer
export LOADBALANCER_VIP=$(openstack floating ip list --port $(openstack loadbalancer show $LOADBALANCER_NAME -f value -c VIP_PORT_ID) -f value -c "Floating IP Address")
# Get the Public IPs of the control plane nodes
for i in $( seq 1 3 ); do
export CP${i}_IP=$(openstack floating ip list --port $(openstack port list --server talos-control-plane-$i -f value -c ID) -f value -c "Floating IP Address")
done
```
Generate the cluster configuration file, substituting the Load Balancer VIP and control plane IPs as additional SANs for the API certificate.
```bash theme={null}
talosctl gen config talos-ovhcloud https://$LOADBALANCER_VIP:6443 \
--additional-sans $CP1_IP,$CP2_IP,$CP3_IP
```
## Step 5. Applying configuration and bootstrapping the cluster
With our configuration file generated, we can now apply it to our control plane nodes and bootstrap the cluster.
```bash theme={null}
# Apply the configuration to the control plane nodes, substituting the Public IPs of the control plane nodes.
for i in $(seq 1 3); do
CP_NODE="CP${i}_IP"
talosctl apply-config -f controlplane.yaml --insecure -n "${!CP_NODE}"
done
# Bootstrap the cluster using one of the control plane nodes
talosctl bootstrap -n $CP1_IP -e $CP1_IP --talosconfig talosconfig
```
Export the `TALOSCONFIG` environment variable to point to the `talosconfig` file. This will allow you to interact with your cluster using `talosctl` without having to specify the `--talosconfig` flag every time.
```bash theme={null}
export TALOSCONFIG=$(pwd)/talosconfig
```
Update the talosconfig file with the endpoints and node information for one of the control plane nodes:
```bash theme={null}
for i in $(seq 1 3); do
CP_NODE="CP${i}_IP"
talosctl config endpoint "${!CP_NODE}"
done
talosctl config node $CP1_IP
```
You should now be able to interact with your cluster with `talosctl`.
## Step 6. Retrieve the `kubeconfig`
At this point we can retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl kubeconfig .
```
## Step 7. Add worker nodes to the cluster
```bash theme={null}
# Create worker node with the user data file
openstack server create talos-worker-1 --flavor $COMPUTE_FLAVOR_NAME --nic net-id=$PRIVATE_NETWORK_ID --image $IMAGE_NAME --user-data /path/to/worker.yaml
```
# Scaleway
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/cloud-platforms/scaleway
Creating a single-instance cluster via the CLI (scw) on scaleway.com.
The process to run a Talos cluster, on a single node in Scaleway is as follows:
## Prerequisites
* Enable block storage on your Scaleway account (Scaleway will only allow snapshots from their block storage, not URLs)
* Configure the `scw` CLI to access your account (optional - you can use the console instead)
* Have `qemu-img` and `wget` installed for image conversion
## Image preparation
1. **Download the image disk** of the Talos version you wish to run:
> You can create your own brew on [Talos Factory](https://factory.talos.dev) if you need a custom image.
> A working extra kernel command line arguments is as follow : `-console talos.platform=scaleway console=ttyS0,115200 talos.dashboard.disabled=0`
2. **Decompress and convert the image**:
```bash theme={null}
zstd --decompress scaleway-amd64.raw.zst
qemu-img convert -O qcow2 scaleway-amd64.raw scaleway-amd64.qcow2
```
3. **Create an S3 bucket** (if it doesn't exist):
Go to Object Storage in the Scaleway console and create a new bucket.
![Scaleway Console Create Bucket]()
![Scaleway Console Create Bucket]()
4. **Upload to S3-compatible object storage**:
Use the Scaleway console Object Storage interface to upload the QCOW2 file directly.
## Snapshot creation
1. Go to the Scaleway Web console.
2. Go to the QCOW2 object in your S3 bucket.
3. Click on the extra action *Import object as snapshot*.
![Scaleway Console Import As Snapshot]()
4. Name the snapshot *scaleway-amd64-v*, and use a Local Storage snapshot type.
## Instance deployment
1. Create instance using the snapshot/image via GUI, CLI, or Infrastructure as Code tools.
2. Open the Security Group in order to open `TCP/50000` and `TCP/6443`.
3. The Talos disk image used to boot is 4.5GB, and is not enough for the EPHEMERAL partition.
4. Add an extra Disk of 25GB (Local Storage) for Talos EPHEMERAL partition. The Talos OS (4.5GB) will be on `/dev/vda` and the extra disk for EPHEMERAL partition will be on `/dev/vdb`. At this point, you can debug and communicate with the instance on --insecure mode :
```
VM_IP=
talosctl -n $VM_IP version --insecure
talosctl -n $VM_IP get disks --insecure
talosctl -n $VM_IP get links --insecure
```
## Talos configuration
As any other Talos instance, generate the Talos machineconfig, with the following patch :
```
# controlplane.patch.yaml
machine:
install:
disk: /dev/vdb
wipe: true
network:
interfaces:
- interface: eth0
dhcp: true
cluster:
allowSchedulingOnControlPlanes: true
apiServer:
extraArgs:
"advertise-address": "0.0.0.0"
apiServer:
certSANs:
- 127.0.0.1
-
```
Generate the Talos config :
```
talosctl gen config talos-cluster https://$VM_IP:6443 --config-patch=@controlplane.patch.yaml --output-dir _out/ --with-docs=false --with-examples=false
```
Also add a VolumeConfig for the EPHEMERAL partition of Talos :
```
---
apiVersion: v1alpha1
kind: VolumeConfig
name: EPHEMERAL
provisioning:
diskSelector:
match: disk.dev_path == '/dev/vdb'
minSize: 2GB
maxSize: 40GB
grow: false
```
Apply the machineconfig + the VolumeConfig on your Talos Scaleway instance.
```
talosctl --nodes $VM_IP apply-config --file _out/controlplane.yaml --insecure
```
At this point, you should be able to communicate with your Talos instance with the right certificates, and debug the Talos instance :
```
talosctl --talosconfig=./_out/talosconfig --nodes $VM_IP -e $VM_IP version
talosctl --talosconfig=./_out/talosconfig --nodes $VM_IP -e $VM_IP dashboard
```
## Talos cluster bootstrap
One last command to bootstrap your Talos Cluster :
```
talosctl --talosconfig=./_out/talosconfig -e $VM_IP -n $VM_IP bootstrap
```
You can follow the logs with the dashboard, and after couple of minutes, your first Talos standalone single instance cluster on Scaleway will be ready to use.
## Notes
* Because the STATE partition will be written to `/dev/vda` during the installation, the instance works correctly with Scaleway's reboot functionality
* `talosctl reset` performs the reset operation but don't start the maintenance mode automatically
* If you give a bigger disk (let's say 25GB) for the Talos disk image for the Scaleway Instance, unfortunately Talos won't be able to expand its volume to 25GB and still see 4.5GB which is not enough for the EPHEMERAL partition
# UpCloud
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/cloud-platforms/upcloud
Creating a cluster via the CLI (upctl) on UpCloud.com.
In this guide we will create an HA Kubernetes cluster 3 control plane nodes and 1 worker node.
We assume some familiarity with UpCloud.
If you need more information on UpCloud specifics, please see the [official UpCloud documentation](https://upcloud.com/resources/docs).
## Create the image
The best way to create an image for UpCloud, is to build one using
[Hashicorp packer](https://www.packer.io/docs/builders/hetzner-cloud), with the
`upcloud-amd64.raw.xz` image available from the [Image Factory](https://factory.talos.dev/).
Using the general ISO is also possible, but the UpCloud image has some UpCloud
specific features implemented, such as the fetching of metadata and user data to configure the nodes.
To create the cluster, you need a few things locally installed:
1. [UpCloud CLI](https://github.com/UpCloudLtd/upcloud-cli)
2. [Hashicorp Packer](https://learn.hashicorp.com/tutorials/packer/get-started-install-cli)
> NOTE: Make sure your account allows API connections.
> To do so, log into
> [UpCloud control panel](https://hub.upcloud.com/login) and go to **People**
> -> **Account** -> **Permissions** -> **Allow API connections** checkbox.
> It is recommended
> to create a separate subaccount for your API access and *only* set the API permission.
To use the UpCloud CLI, you need to create a config in `$HOME/.config/upctl.yaml`
```yaml theme={null}
username: your_upcloud_username
password: your_upcloud_password
```
To use the UpCloud packer plugin, you need to also export these credentials to your
environment variables, by e.g. putting the following in your `.bashrc` or `.zshrc`
```shell theme={null}
export UPCLOUD_USERNAME=""
export UPCLOUD_PASSWORD=""
```
Next create a config file for packer to use:
Now create a new image by issuing the commands shown below.
```bash theme={null}
packer init .
packer build .
```
After doing this, you can find the custom image in the console interface under storage.
## Creating a cluster via the CLI
### Create an endpoint
To communicate with the Talos cluster you will need a single endpoint that is used
to access the cluster.
This can either be a loadbalancer that will sit in front of
all your control plane nodes, a DNS name with one or more A or AAAA records pointing
to the control plane nodes, or directly the IP of a control plane node.
Which option is best for you will depend on your needs.
Endpoint selection has been further documented [here](../../getting-started/getting-started#decide-the-kubernetes-endpoint).
After you decide on which endpoint to use, note down the domain name or IP, as
we will need it in the next step.
### Create the machine configuration files
#### Generating base configurations
Using the DNS name of the endpoint created earlier, generate the base
configuration files for the Talos machines:
```bash theme={null}
$ talosctl gen config talos-upcloud-tutorial https://: --install-disk /dev/vda
created controlplane.yaml
created worker.yaml
created talosconfig
```
At this point, you can modify the generated configs to your liking.
Depending on the Kubernetes version you want to run, you might need to select a different Talos version, as not all versions are compatible.
You can find the support matrix [here](../../getting-started/support-matrix).
Optionally, you can specify [machine configuration patches](../../configure-your-talos-cluster/system-configuration/patching#configuration-patching-with-talosctl-cli)
which will be applied during the config generation.
#### Validate the configuration files
```bash theme={null}
$ talosctl validate --config controlplane.yaml --mode cloud
controlplane.yaml is valid for cloud mode
$ talosctl validate --config worker.yaml --mode cloud
worker.yaml is valid for cloud mode
```
### Create the servers
#### Create the control plane nodes
Run the following to create three total control plane nodes:
> Note: modify the zone and OS depending on your preferences.
> The OS should match the template name generated with packer in the previous step.
Note the IP address of the first control plane node, as we will need it later.
#### Create the worker nodes
Run the following to create a worker node:
### Bootstrap etcd
To configure `talosctl` we will need the first control plane node's IP, as noted earlier.
We only add one node IP, as that is the entry into our cluster against which our commands will be run.
All requests to other nodes are proxied through the endpoint, and therefore not
all nodes need to be manually added to the config.
You don't want to run your commands against all nodes, as this can destroy your
cluster if you are not careful [(further documentation)](../../getting-started/getting-started#configure-your-talosctl-client).
Set the `endpoints` and `nodes`:
```bash theme={null}
talosctl --talosconfig talosconfig config endpoint
talosctl --talosconfig talosconfig config node
```
Bootstrap `etcd`:
```bash theme={null}
talosctl --talosconfig talosconfig bootstrap
```
### Retrieve the `kubeconfig`
At this point we can retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl --talosconfig talosconfig kubeconfig
```
It will take a few minutes before Kubernetes has been fully bootstrapped, and is accessible.
You can check if the nodes are registered in Talos by running
```bash theme={null}
talosctl --talosconfig talosconfig get members
```
To check if your nodes are ready, run
```bash theme={null}
kubectl get nodes
```
# Vultr
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/cloud-platforms/vultr
Creating a cluster via the CLI (vultr-cli) on Vultr.com.
## Creating a Cluster using the Vultr CLI
This guide will demonstrate how to create a highly-available Kubernetes cluster with one worker using the Vultr cloud provider.
[Vultr](https://www.vultr.com/) have a very well documented REST API, and an open-source [CLI](https://github.com/vultr/vultr-cli) tool to interact with the API which will be used in this guide.
Make sure to follow installation and authentication instructions for the `vultr-cli` tool.
### Boot options
#### Upload an ISO image
First step is to make the Talos ISO available to Vultr by uploading the latest release of the ISO to the Vultr ISO server.
Make a note of the `ID` in the output, it will be needed later when creating the instances.met
#### PXE booting via image factory
Talos Linux can be PXE-booted on Vultr using [Image Factory](../../learn-more/image-factory), using the `vultr` platform: e.g.
(this URL references the default schematic and `amd64` architecture).
Make a note of the `ID` in the output, it will be needed later when creating the instances.
### Create a load balancer
A load balancer is needed to serve as the Kubernetes endpoint for the cluster.
```bash theme={null}
vultr-cli load-balancer create \
--region $REGION \
--label "Talos Kubernetes Endpoint" \
--port 6443 \
--protocol tcp \
--check-interval 10 \
--response-timeout 5 \
--healthy-threshold 5 \
--unhealthy-threshold 3 \
--forwarding-rules frontend_protocol:tcp,frontend_port:443,backend_protocol:tcp,backend_port:6443
```
Make a note of the `ID` of the load balancer from the output of the above command, it will be needed after the control plane instances are created.
```bash theme={null}
vultr-cli load-balancer get $LOAD_BALANCER_ID | grep ^IP
```
Make a note of the `IP` address, it will be needed later when generating the configuration.
### Create the machine configuration
#### Generate base configuration
Using the IP address (or DNS name if one was created) of the load balancer created above, generate the machine configuration files for the new cluster.
```bash theme={null}
talosctl gen config talos-kubernetes-vultr https://$LOAD_BALANCER_ADDRESS
```
Once generated, the machine configuration can be modified as necessary for the new cluster, for instance updating disk installation, or adding SANs for the certificates.
#### Validate the configuration files
```bash theme={null}
talosctl validate --config controlplane.yaml --mode cloud
talosctl validate --config worker.yaml --mode cloud
```
### Create the nodes
#### Create the control plane nodes
First a control plane needs to be created, with the example below creating 3 instances in a loop.
The instance type (noted by the `--plan vc2-2c-4gb` argument) in the example is for a minimum-spec control plane node, and should be updated to suit the cluster being created.
```bash theme={null}
for id in $(seq 3); do
vultr-cli instance create \
--plan vc2-2c-4gb \
--region $REGION \
--iso $TALOS_ISO_ID \
--host talos-k8s-cp${id} \
--label "Talos Kubernetes Control Plane" \
--tags talos,kubernetes,control-plane
done
```
Make a note of the instance `ID`s, as they are needed to attach to the load balancer created earlier.
```bash theme={null}
vultr-cli load-balancer update $LOAD_BALANCER_ID --instances $CONTROL_PLANE_1_ID,$CONTROL_PLANE_2_ID,$CONTROL_PLANE_3_ID
```
Once the nodes are booted and waiting in maintenance mode, the machine configuration can be applied to each one in turn.
```bash theme={null}
talosctl --talosconfig talosconfig apply-config --insecure --nodes $CONTROL_PLANE_1_ADDRESS --file controlplane.yaml
talosctl --talosconfig talosconfig apply-config --insecure --nodes $CONTROL_PLANE_2_ADDRESS --file controlplane.yaml
talosctl --talosconfig talosconfig apply-config --insecure --nodes $CONTROL_PLANE_3_ADDRESS --file controlplane.yaml
```
#### Create the worker nodes
Now worker nodes can be created and configured in a similar way to the control plane nodes, the difference being mainly in the machine configuration file.
Note that like with the control plane nodes, the instance type (here set by `--plan vc2-1-1gb`) should be changed for the actual cluster requirements.
```bash theme={null}
for id in $(seq 1); do
vultr-cli instance create \
--plan vc2-1c-1gb \
--region $REGION \
--iso $TALOS_ISO_ID \
--host talos-k8s-worker${id} \
--label "Talos Kubernetes Worker" \
--tags talos,kubernetes,worker
done
```
Once the worker is booted and in maintenance mode, the machine configuration can be applied in the following manner.
```bash theme={null}
talosctl --talosconfig talosconfig apply-config --insecure --nodes $WORKER_1_ADDRESS --file worker.yaml
```
### Bootstrap etcd
Once all the cluster nodes are correctly configured, the cluster can be bootstrapped to become functional.
It is important that the `talosctl bootstrap` command be executed only once and against only a single control plane node.
```bash theme={null}
talosctl --talosconfig talosconfig bootstrap --endpoints $CONTROL_PLANE_1_ADDRESS --nodes $CONTROL_PLANE_1_ADDRESS
```
### Configure endpoints and nodes
While the cluster goes through the bootstrapping process and beings to self-manage, the `talosconfig` can be updated with the [endpoints and nodes](../../learn-more/talosctl#endpoints-and-nodes).
```bash theme={null}
talosctl --talosconfig talosconfig config endpoints $CONTROL_PLANE_1_ADDRESS $CONTROL_PLANE_2_ADDRESS $CONTROL_PLANE_3_ADDRESS
talosctl --talosconfig talosconfig config nodes $CONTROL_PLANE_1_ADDRESS $CONTROL_PLANE_2_ADDRESS $CONTROL_PLANE_3_ADDRESS WORKER_1_ADDRESS
```
### Retrieve the `kubeconfig`
Finally, with the cluster fully running, the administrative `kubeconfig` can be retrieved from the Talos API to be saved locally.
```bash theme={null}
talosctl --talosconfig talosconfig kubeconfig .
```
Now the `kubeconfig` can be used by any of the usual Kubernetes tools to interact with the Talos-based Kubernetes cluster as normal.
# Docker
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/local-platforms/docker
Creating Talos Kubernetes cluster using Docker.
In this guide we will create a Kubernetes cluster in Docker, using a containerized version of Talos.
Running Talos in Docker is intended to be used in CI pipelines, and local testing when you need a quick and easy cluster.
Furthermore, if you are running Talos in production, it provides an excellent way for developers to develop against the same version of Talos.
## Requirements
The follow are requirements for running Talos in Docker:
* Docker 18.03 or greater
* a recent version of [`talosctl`](https://github.com/siderolabs/talos/releases)
If you are using Docker Desktop on a macOS computer, and you encounter the error: `Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?` You may need to manually create the link for the Docker socket:
`sudo ln -s "$HOME/.docker/run/docker.sock" /var/run/docker.sock`.
## Caveats
Due to the fact that Talos will be running in a container, certain APIs are not available.
For example `upgrade`, `reset`, and similar APIs don't apply in container mode.
Further, when running on a Mac in docker, due to networking limitations, VIPs are not supported.
## Create the cluster
Creating a local cluster is as simple as:
```bash theme={null}
talosctl cluster create docker
```
Once the above finishes successfully, your `talosconfig` (`~/.talos/config`) and `kubeconfig` (`~/.kube/config`) will be configured to point to the new cluster.
> Note: Startup times can take up to a minute or more before the cluster is available.
Finally, we just need to specify which nodes you want to communicate with using `talosctl`.
Talosctl can operate on one or all the nodes in the cluster – this makes cluster wide commands much easier.
`talosctl config nodes 10.5.0.2 10.5.0.3`
Talos and Kubernetes API are mapped to a random port on the host machine, the retrieved `talosconfig` and `kubeconfig` are configured automatically to point to the new cluster.
Talos API endpoint can be found using `talosctl config info`:
```bash theme={null}
$ talosctl config info
...
Endpoints: 127.0.0.1:38423
```
Kubernetes API endpoint is available with `talosctl cluster show`:
```bash theme={null}
$ talosctl cluster show
...
KUBERNETES ENDPOINT https://127.0.0.1:43083
```
> **Note:** When running Kubernetes with Flannel inside Docker, you may encounter this error:
> `Failed to check br_netfilter: stat /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory`
>
> This happens because the host Linux kernel does not have the **`br_netfilter`** module enabled.
> To resolve this, load the module by running
>
> ```bash theme={null}
> sudo modprobe br_netfilter
> ```
## Using the cluster
Once the cluster is available, you can make use of `talosctl` and `kubectl` to interact with the cluster.
For example, to view current running containers, run `talosctl containers` for a list of containers in the `system` namespace, or `talosctl containers -k` for the `k8s.io` namespace.
To view the logs of a container, use `talosctl logs ` or `talosctl logs -k `.
## Cleaning Up
To cleanup, run:
```bash theme={null}
talosctl cluster destroy
```
## Multiple clusters
Multiple Talos Linux cluster can be created on the same host, each cluster will need to have:
* a unique name (default is `talos-default`)
* a unique network CIDR (default is `10.5.0.0/24`)
To create a new cluster, run:
```bash theme={null}
talosctl cluster create --name cluster2 --cidr 10.6.0.0/24
```
To destroy a specific cluster, run:
```bash theme={null}
talosctl cluster destroy --name cluster2
```
To switch between clusters, use `--context` flag:
```bash theme={null}
talosctl --context cluster2 version
kubectl --context admin@cluster2 get nodes
```
## Running Talos in Docker manually
To run Talos in a container manually, run:
The machine configuration submitted to the container should have a [host DNS feature](../../reference/configuration/v1alpha1/config#Config.machine.features.hostDNS) enabled with `forwardKubeDNSToHost` enabled.
It is used to forward DNS requests to the resolver provided by Docker (or other container runtime).
# QEMU
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/local-platforms/qemu
Creating Talos Kubernetes cluster using QEMU VMs.
In this guide we will create a Kubernetes cluster using QEMU.
![]()
## Requirements
* QEMU
* a kernel with
* KVM enabled (`/dev/kvm` must exist)
* `CONFIG_NET_SCH_NETEM` enabled
* `CONFIG_NET_SCH_INGRESS` enabled
* at least `CAP_SYS_ADMIN` and `CAP_NET_ADMIN` capabilities
* `bridge`, `static` and `firewall` CNI plugins from the [standard CNI plugins](https://github.com/containernetworking/cni), and `tc-redirect-tap` CNI plugin from the [awslabs tc-redirect-tap](https://github.com/awslabs/tc-redirect-tap) installed to `/opt/cni/bin` (installed automatically by `talosctl`)
* iptables
* `/var/run/netns` directory should exist
* QEMU
* Apple Silicon Mac
## Installation
### How to get QEMU
Install QEMU with your operating system package manager.
For example, on Ubuntu for x86:
```bash theme={null}
apt install qemu-system-x86 qemu-kvm
```
Via brew:
```bash theme={null}
brew install qemu
```
### Install talosctl
You can download `talosctl` on MacOS and Linux via:
```bash theme={null}
brew install siderolabs/tap/talosctl
```
For manual installation and other platforms please see the [talosctl installation guide](../../getting-started/talosctl).
## Create the cluster
For the first time, create root state directory as your user so that you can inspect the logs as non-root user:
```bash theme={null}
mkdir -p ~/.talos/clusters
```
Create the cluster:
```bash theme={null}
sudo --preserve-env=HOME talosctl cluster create qemu
```
On Linux, before the first cluster is created, `talosctl` will download the CNI bundle for the VM provisioning and install it to `~/.talos/cni` directory.
Once the above finishes successfully, your talosconfig (`~/.talos/config`) will be configured to point to the new cluster, and `kubeconfig` will be
downloaded and merged into default kubectl config location (`~/.kube/config`).
### Presets
The `talosctl cluster create qemu` command has various presets available via the `--presets` flag to modify behavior.
See [reference documentation](../../reference/cli#talosctl-cluster-create-qemu") for more details.
### Image factory schematic
Optionally a custom [Image Factory](../../getting-started/talosctl) schematic ID can be provided via the `--schematic-id` flag.
### Omni
The `omni-api-endpoint` flag configures nodes to connect to an Omni instance once booted.
Using [SideroLink](../../networking/siderolink), the local QEMU nodes can communicate with Omni as long as the endpoint is reachable.
This enables connections to a local Omni instance, a cloud-hosted Omni instance, or a Sidero SaaS Omni instance.
## Using the cluster
Once the cluster is available, you can make use of `talosctl` and `kubectl` to interact with the cluster.
For example, to view current running containers, run `talosctl -n 10.5.0.2 containers` for a list of containers in the `system` namespace, or `talosctl -n 10.5.0.2 containers -k` for the `k8s.io` namespace.
To view the logs of a container, use `talosctl -n 10.5.0.2 logs ` or `talosctl -n 10.5.0.2 logs -k `.
A bridge interface will be created, and assigned the default IP 10.5.0.1.
Each node will be directly accessible on the subnet specified at cluster creation time.
A loadbalancer runs on 10.5.0.1 by default, which handles loadbalancing for the Kubernetes APIs.
You can see a summary of the cluster state by running:
```bash theme={null}
$ talosctl cluster show --provisioner qemu
PROVISIONER qemu
NAME talos-default
NETWORK NAME talos-default
NETWORK CIDR 10.5.0.0/24
NETWORK GATEWAY 10.5.0.1
NETWORK MTU 1500
NODES:
NAME TYPE IP CPU RAM DISK
talos-default-controlplane-1 ControlPlane 10.5.0.2 1.00 1.6 GB 4.3 GB
talos-default-controlplane-2 ControlPlane 10.5.0.3 1.00 1.6 GB 4.3 GB
talos-default-controlplane-3 ControlPlane 10.5.0.4 1.00 1.6 GB 4.3 GB
talos-default-worker-1 Worker 10.5.0.5 1.00 1.6 GB 4.3 GB
```
## Cleaning up
To cleanup, run:
```bash theme={null}
sudo --preserve-env=HOME talosctl cluster destroy --provisioner qemu
```
> **Note**: In that case that the host machine is rebooted before destroying the cluster, you may need to manually remove `~/.talos/clusters/talos-default`.
## Manual clean up
The `talosctl cluster destroy` command depends heavily on the clusters state directory.
It contains all related information of the cluster.
The PIDs and network associated with the cluster nodes.
If you happened to have deleted the state folder by mistake or you would like to cleanup
the environment, here are the steps how to do it manually:
### Remove VM launchers
Find the process of `talosctl qemu-launch`:
```bash theme={null}
ps -elf | grep 'talosctl qemu-launch'
```
To remove the VMs manually, execute:
```bash theme={null}
sudo kill -s SIGTERM
```
Example output, where VMs are running with PIDs **157615** and **157617**
```bash theme={null}
ps -elf | grep '[t]alosctl qemu-launch'
0 S root 157615 2835 0 80 0 - 184934 - 07:53 ? 00:00:00 talosctl qemu-launch
0 S root 157617 2835 0 80 0 - 185062 - 07:53 ? 00:00:00 talosctl qemu-launch
sudo kill -s SIGTERM 157615
sudo kill -s SIGTERM 157617
```
### Stopping VMs
Find the process of `qemu-system`:
```bash theme={null}
ps -elf | grep 'qemu-system'
```
To stop the VMs manually, execute:
```bash theme={null}
sudo kill -s SIGTERM
```
Example output, where VMs are running with PIDs **158065** and **158216**
```bash theme={null}
ps -elf | grep qemu-system
2 S root 1061663 1061168 26 80 0 - 1786238 - 14:05 ? 01:53:56 qemu-system-x86_64 -m 2048 -drive format=raw,if=virtio,file=/home/username/.talos/clusters/talos-default/bootstrap-master.disk -smp cpus=2 -cpu max -nographic -netdev tap,id=net0,ifname=tap0,script=no,downscript=no -device virtio-net-pci,netdev=net0,mac=1e:86:c6:b4:7c:c4 -device virtio-rng-pci -no-reboot -boot order=cn,reboot-timeout=5000 -smbios type=1,uuid=7ec0a73c-826e-4eeb-afd1-39ff9f9160ca -machine q35,accel=kvm
2 S root 1061663 1061170 67 80 0 - 621014 - 21:23 ? 00:00:07 qemu-system-x86_64 -m 2048 -drive format=raw,if=virtio,file=/homeusername/.talos/clusters/talos-default/pxe-1.disk -smp cpus=2 -cpu max -nographic -netdev tap,id=net0,ifname=tap0,script=no,downscript=no -device virtio-net-pci,netdev=net0,mac=36:f3:2f:c3:9f:06 -device virtio-rng-pci -no-reboot -boot order=cn,reboot-timeout=5000 -smbios type=1,uuid=ce12a0d0-29c8-490f-b935-f6073ab916a6 -machine q35,accel=kvm
sudo kill -s SIGTERM 1061663
sudo kill -s SIGTERM 1061663
```
### Remove load balancer
Find the process of `talosctl loadbalancer-launch`:
```bash theme={null}
ps -elf | grep 'talosctl loadbalancer-launch'
```
To remove the LB manually, execute:
```bash theme={null}
sudo kill -s SIGTERM
```
Example output, where loadbalancer is running with PID **157609**
```bash theme={null}
ps -elf | grep '[t]alosctl loadbalancer-launch'
4 S root 157609 2835 0 80 0 - 184998 - 07:53 ? 00:00:07 talosctl loadbalancer-launch --loadbalancer-addr 10.5.0.1 --loadbalancer-upstreams 10.5.0.2
sudo kill -s SIGTERM 157609
```
### Remove DHCP server
Find the process of `talosctl dhcpd-launch`:
```bash theme={null}
ps -elf | grep 'talosctl dhcpd-launch'
```
To remove the LB manually, execute:
```bash theme={null}
sudo kill -s SIGTERM
```
Example output, where loadbalancer is running with PID **157609**
```bash theme={null}
ps -elf | grep '[t]alosctl dhcpd-launch'
4 S root 157609 2835 0 80 0 - 184998 - 07:53 ? 00:00:07 talosctl dhcpd-launch --state-path /home/username/.talos/clusters/talos-default --addr 10.5.0.1 --interface talosbd9c32bc
sudo kill -s SIGTERM 157609
```
### Remove network
This is more tricky part as if you have already deleted the state folder.
If you didn't then it is written in the `state.yaml` in the
`~/.talos/clusters/` directory.
```bash theme={null}
sudo cat ~/.talos/clusters//state.yaml | grep bridgename
bridgename: talos
```
If you only had one cluster, then it will be the interface with name
`talos`
```bash theme={null}
46: talos: mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether a6:72:f4:0a:d3:9c brd ff:ff:ff:ff:ff:ff
inet 10.5.0.1/24 brd 10.5.0.255 scope global talos17c13299
valid_lft forever preferred_lft forever
inet6 fe80::a472:f4ff:fe0a:d39c/64 scope link
valid_lft forever preferred_lft forever
```
To remove this interface:
```bash theme={null}
sudo ip link del talos
```
The bridge is automatically created by qemu, so removing all the machines will also result in the deletion of the bridge interface.
### Remove state directory
To remove the state directory execute:
```bash theme={null}
sudo rm -Rf /home/$USER/.talos/clusters/
```
## Troubleshooting
### Logs
Inspect logs directory
```bash theme={null}
sudo cat ~/.talos/clusters//*.log
```
Logs are saved under `--.log`
For example in case of **k8s** cluster name:
```bash theme={null}
ls -la ~/.talos/clusters/k8s | grep log
-rw-r--r--. 1 root root 69415 Apr 26 20:58 k8s-master-1.log
-rw-r--r--. 1 root root 68345 Apr 26 20:58 k8s-worker-1.log
-rw-r--r--. 1 root root 24621 Apr 26 20:59 lb.log
```
Inspect logs during the installation
```bash theme={null}
tail -f ~/.talos/clusters//*.log
```
## How it works
### Linux
On Linux the KVM accelerator is utilized.
Networking is created using the CNI plugins.
For DHCP a custom server is used.
### MacOS
On Macos the `hvf` accelerator (Apple Hypervisor Framework) is utilized.
Networking is created by QEMU via the [apple vmnet framework](https://developer.apple.com/documentation/vmnet) which handles everything but the DHCP for which a custom server is used instead.
# VirtualBox
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/local-platforms/virtualbox
Creating Talos Kubernetes cluster using VirtualBox VMs.
In this guide we will create a Kubernetes cluster using VirtualBox.
## Video walkthrough
To see a live demo of this writeup, visit Youtube here:
## Installation
### How to get VirtualBox
Install VirtualBox with your operating system package manager or from the [website](https://www.virtualbox.org/).
For example, on Ubuntu for x86:
```bash theme={null}
apt install virtualbox
```
### Install talosctl
You can download `talosctl` on MacOS and Linux via:
```bash theme={null}
brew install siderolabs/tap/talosctl
```
For manual installation and other platforms please see the [talosctl installation guide](../../getting-started/talosctl).
### Download ISO image
Download the ISO image from [Image Factory](https://www.talos.dev/latest/talos-guides/install/boot-assets/#image-factory).
```bash theme={null}
mkdir -p _out/
curl https://factory.talos.dev/image/376567988ad370138ad8b2698212367b8edcb69b5fd68c80be1f2ec7d603b4ba//metal-.iso -L -o _out/metal-.iso
```
For example version \$ for `linux` platform:
## Create VMs
Start by creating a new VM by clicking the "New" button in the VirtualBox UI:
![]()
Supply a name for this VM, and specify the Type and Version:
![]()
> Note: Make sure type is set as `Linux / Other Linux (64-bit)`.
Edit the memory to supply at least 2GB of RAM for the VM:
![]()
Proceed through the disk settings, keeping the defaults.
You can increase the disk space if desired.
Once created, select the VM and hit "Settings":
![]()
In the "System" section, supply at least 2 CPUs:
![]()
In the "Network" section, switch the network "Attached To" section to "Bridged Adapter":
![]()
Finally, in the "Storage" section, select the optical drive and, on the right, select the ISO by browsing your filesystem:
![]()
Repeat this process for a second VM to use as a worker node.
You can also repeat this for additional nodes desired.
## Start control plane node
Once the VMs have been created and updated, start the VM that will be the first control plane node.
This VM will boot the ISO image specified earlier and enter "maintenance mode".
Once the machine has entered maintenance mode, there will be a console log that details the IP address that the node received.
Take note of this IP address, which will be referred to as `$CONTROL_PLANE_IP` for the rest of this guide.
If you wish to export this IP as a bash variable, simply issue a command like `export CONTROL_PLANE_IP=1.2.3.4`.
![]()
## Generate machine configurations
With the IP address above, you can now generate the machine configurations to use for installing Talos and Kubernetes.
Issue the following command, updating the output directory, cluster name, and control plane IP as you see fit:
```bash theme={null}
talosctl gen config talos-vbox-cluster https://$CONTROL_PLANE_IP:6443 --output-dir _out
```
This will create several files in the `_out` directory: controlplane.yaml, worker.yaml, and talosconfig.
## Create control plane node
Using the `controlplane.yaml` generated above, you can now apply this config using talosctl.
Issue:
```bash theme={null}
talosctl apply-config --insecure --nodes $CONTROL_PLANE_IP --file _out/controlplane.yaml
```
You should now see some action in the VirtualBox console for this VM.
Talos will be installed to disk, the VM will reboot, and then Talos will configure the Kubernetes control plane on this VM.
> Note: This process can be repeated multiple times to create an HA control plane.
>
> Note: If you encounter the following error in the console:
>
> `Talos is already installed to disk but booted from another media and talos.halt_if_installed kernel parameter set. Please reboot from another disk.`
>
> Simply remove the ISO image from the VM and restart it.
## Create worker node
Create at least a single worker node using a process similar to the control plane creation above.
Start the worker node VM and wait for it to enter "maintenance mode".
Take note of the worker node's IP address, which will be referred to as `$WORKER_IP`.
If you wish to export this IP as a bash variable, simply issue a command like `export WORKER_IP=1.2.3.4`.
Issue:
```bash theme={null}
talosctl apply-config --insecure --nodes $WORKER_IP --file _out/worker.yaml
```
> Note: This process can be repeated multiple times to add additional workers.
### Bootstrap `etcd`
Before the cluster is ready, the `etcd` has to be bootstrapped.
The cluster will be in stage `Booting` and `healthy` state until this is stage is completed.
Set the `endpoints` and `nodes`:
```bash theme={null}
talosctl --talosconfig $TALOSCONFIG config endpoint
talosctl --talosconfig $TALOSCONFIG config node
```
Bootstrap `etcd` by running the following command.
You should see stage change to `Running` and your cluster is now ready.
```bash theme={null}
talosctl --talosconfig $TALOSCONFIG bootstrap
```
## Using the cluster
Once the cluster is available, you can make use of `talosctl` and `kubectl` to interact with the cluster.
For example, to view current running containers, run `talosctl containers` for a list of containers in the `system` namespace, or `talosctl containers -k` for the `k8s.io` namespace.
To view the logs of a container, use `talosctl logs ` or `talosctl logs -k `.
First, configure talosctl to talk to your control plane node by issuing the following, updating paths and IPs as necessary:
```bash theme={null}
export TALOSCONFIG="_out/talosconfig"
talosctl config endpoint $CONTROL_PLANE_IP
talosctl config node $CONTROL_PLANE_IP
```
### Retrieve the `kubeconfig`
At this point we can retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl --talosconfig $TALOSCONFIG kubeconfig .
```
Export the config so kubectl can find it: `export KUBECONFIG=$(pwd)/kubeconfig`.
You can then use kubectl in this fashion:
```bash theme={null}
kubectl get nodes
```
## Cleaning up
To cleanup, simply stop and delete the virtual machines from the VirtualBox UI.
# Omni SaaS
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/omni
Omni is a project created by the Talos team that has native support for Talos Linux.
Omni allows you to start with bare metal, virtual machines or a cloud provider, and create clusters spanning all of your locations, with a few clicks.
You provide the machines – edge compute, bare metal, VMs, or in your cloud account.
Boot from an Omni Talos Linux image.
Click to allocate to a cluster.
That’s it!
* Vanilla Kubernetes, on your machines, under your control.
* Elegant UI for management and operations
* Security taken care of – ties into your Enterprise ID provider
* Highly Available Kubernetes API end point built in
* Firewall friendly: manage Edge nodes securely
* From single-node clusters to the largest scale
* Support for GPUs and most CSIs.
The Omni SaaS is available to run locally, to support air-gapped security and data sovereignty concerns.
Omni handles the lifecycle of Talos Linux machines, provides unified access to the Talos and Kubernetes API tied to the identity provider of your choice,
and provides a UI for cluster management and operations.
Omni automates scaling the clusters up and down, and provides a unified view of the state of your clusters.
See more in the [Omni documentation](https://www.siderolabs.com/platform/saas-for-kubernetes/).
# Banana Pi M64
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/single-board-computers/bananapi_m64
Installing Talos on Banana Pi M64 SBC using raw disk image.
## Prerequisites
You will need
* `talosctl`
* an SD card
Download the latest `talosctl`.
## Download the image using Image Factory
The default schematic id for "vanilla" Banana Pi M64 is `8e11dcb3c2803fbe893ab201fcadf1ef295568410e7ced95c6c8b122a5070ce4`.
Refer to the [Image Factory](../../learn-more/image-factory) documentation for more information.
Download the image and decompress it:
## Writing the image
The path to your SD card can be found using `fdisk` on Linux or `diskutil` on macOS.
In this example, we will assume `/dev/mmcblk0`.
Now `dd` the image to your SD card:
```bash theme={null}
sudo dd if=metal-arm64.raw of=/dev/mmcblk0 conv=fsync bs=4M
```
## Bootstrapping the node
Insert the SD card to your board, turn it on and wait for the console to show you the instructions for bootstrapping the node.
Following the instructions in the console output to connect to the interactive installer:
```bash theme={null}
talosctl apply-config --insecure --mode=interactive --nodes
```
Once the interactive installation is applied, the cluster will form and you can then use `kubectl`.
## Retrieve the `kubeconfig`
Retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl kubeconfig
```
## Upgrading
For example, to upgrade to the latest version of Talos, you can run:
# Jetson Nano
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/single-board-computers/jetson_nano
Installing Talos on Jetson Nano SBC using raw disk image.
## Prerequisites
You will need
* `talosctl`
* an SD card/USB drive
* [crane CLI](https://github.com/google/go-containerregistry/releases)
Download the latest `talosctl`.
## Flashing the firmware to on-board SPI flash
> Flashing the firmware only needs to be done once.
We will use the [R32.7.2 release](https://developer.nvidia.com/embedded/l4t/r32_release_v7.1/t210/jetson-210_linux_r32.7.2_aarch64.tbz2) for the Jetson Nano.
Most of the instructions is similar to this [doc](https://nullr0ute.com/2020/11/installing-fedora-on-the-nvidia-jetson-nano/) except that we'd be using a upstream version of `u-boot` with patches from NVIDIA u-boot so that USB boot also works.
Before flashing we need the following:
* A USB-A to micro USB cable
* A jumper wire to enable recovery mode
* A HDMI monitor to view the logs if the USB serial adapter is not available
* A USB to Serial adapter with 3.3V TTL (optional)
* A 5V DC barrel jack
If you're planning to use the serial console follow the documentation [here](https://www.jetsonhacks.com/2019/04/19/jetson-nano-serial-console/)
First start by downloading the Jetson Nano L4T release.
```bash theme={null}
curl -SLO https://developer.nvidia.com/embedded/l4t/r32_release_v7.1/t210/jetson-210_linux_r32.7.2_aarch64.tbz2
```
Next we will extract the L4T release and replace the `u-boot` binary with the patched version.
```bash theme={null}
tar xf jetson-210_linux_r32.6.1_aarch64.tbz2
cd Linux_for_Tegra
crane --platform=linux/arm64 export ghcr.io/siderolabs/sbc-jetson:v0.1.0 - | tar xf - --strip-components=4 -C bootloader/t210ref/p3450-0000/ artifacts/arm64/u-boot/jetson_nano/u-boot.bin
```
Next we will flash the firmware to the Jetson Nano SPI flash.
In order to do that we need to put the Jetson Nano into Force Recovery Mode (FRC).
We will use the instructions from [here](https://developer.download.nvidia.com/embedded/L4T/r32_Release_v4.4/r32_Release_v4.4-GMC3/T210/l4t_quick_start_guide.txt)
* Ensure that the Jetson Nano is powered off.
There is no need for the SD card/USB storage/network cable to be connected
* Connect the micro USB cable to the micro USB port on the Jetson Nano, don't plug the other end to the PC yet
* Enable Force Recovery Mode (FRC) by placing a jumper across the FRC pins on the Jetson Nano
* For board revision *A02*, these are pins `3` and `4` of header `J40`
* For board revision *B01*, these are pins `9` and `10` of header `J50`
* Place another jumper across `J48` to enable power from the DC jack and connect the Jetson Nano to the DC jack `J25`
* Now connect the other end of the micro USB cable to the PC and remove the jumper wire from the FRC pins
Now the Jetson Nano is in Force Recovery Mode (FRC) and can be confirmed by running the following command
```bash theme={null}
lsusb | grep -i "nvidia"
```
Now we can move on the flashing the firmware.
```bash theme={null}
sudo ./flash p3448-0000-max-spi external
```
This will flash the firmware to the Jetson Nano SPI flash and you'll see a lot of output.
If you've connected the serial console you'll also see the progress there.
Once the flashing is done you can disconnect the USB cable and power off the Jetson Nano.
## Download the image
The default schematic id for "vanilla" Jetson Nano is `c7d6f36c6bdfb45fd63178b202a67cff0dd270262269c64886b43f76880ecf1e`.
Refer to the [Image Factory](../../learn-more/image-factory) documentation for more information.
Download the image and decompress it:
## Writing the image
Now `dd` the image to your SD card/USB storage:
```bash theme={null}
sudo dd if=metal-arm64.raw of=/dev/mmcblk0 conv=fsync bs=4M status=progress
```
\| Replace `/dev/mmcblk0` with the name of your SD card/USB storage.
## Bootstrapping the node
Insert the SD card/USB storage to your board, turn it on and wait for the console to show you the instructions for bootstrapping the node.
Following the instructions in the console output to connect to the interactive installer:
```bash theme={null}
talosctl apply-config --insecure --mode=interactive --nodes
```
Once the interactive installation is applied, the cluster will form and you can then use `kubectl`.
## Retrieve the `kubeconfig`
Retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl kubeconfig
```
## Upgrading
For example, to upgrade to the latest version of Talos, you can run:
# Libre Computer Board ALL-H3-CC
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/single-board-computers/libretech_all_h3_cc_h5
Installing Talos on Libre Computer Board ALL-H3-CC SBC using raw disk image.
## Prerequisites
You will need
* `talosctl`
* an SD card
Download the latest `talosctl`.
## Download the image
The default schematic id for "vanilla" Libretech H3 CC H5 is `5689d7795f91ac5bf6ccc85093fad8f8b27f6ea9d96a9ac5a059997bffd8ad5c`.
Refer to the [Image Factory](../../learn-more/image-factory) documentation for more information.
Download the image and decompress it:
## Writing the image
The path to your SD card can be found using `fdisk` on Linux or `diskutil` on macOS.
In this example, we will assume `/dev/mmcblk0`.
Now `dd` the image to your SD card:
```bash theme={null}
sudo dd if=metal-arm64.raw of=/dev/mmcblk0 conv=fsync bs=4M
```
## Bootstrapping the node
Insert the SD card to your board, turn it on and wait for the console to show you the instructions for bootstrapping the node.
Create a `installer-patch.yaml` containing reference to the `installer` image generated from an overlay:
Following the instructions in the console output to connect to the interactive installer:
```bash theme={null}
talosctl apply-config --insecure --mode=interactive --nodes
```
Once the interactive installation is applied, the cluster will form and you can then use `kubectl`.
## Retrieve the `kubeconfig`
Retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl kubeconfig
```
## Upgrading
For example, to upgrade to the latest version of Talos, you can run:
# Friendlyelec Nano PI R4S
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/single-board-computers/nanopi_r4s
Installing Talos on a Nano PI R4S SBC using raw disk image.
## Prerequisites
You will need
* `talosctl`
* an SD card
Download the latest `talosctl`.
## Download the image
The default schematic id for "vanilla" NanoPi R4S is `5f74a09891d5830f0b36158d3d9ea3b1c9cc019848ace08ff63ba255e38c8da4`.
Refer to the [Image Factory](../../learn-more/image-factory) documentation for more information.
Download the image and decompress it:
## Writing the image
The path to your SD card can be found using `fdisk` on Linux or `diskutil` on macOS.
In this example, we will assume `/dev/mmcblk0`.
Now `dd` the image to your SD card:
```bash theme={null}
sudo dd if=metal-arm64.raw of=/dev/mmcblk0 conv=fsync bs=4M
```
## Bootstrapping the node
Insert the SD card to your board, turn it on and wait for the console to show you the instructions for bootstrapping the node.
Following the instructions in the console output to connect to the interactive installer:
```bash theme={null}
talosctl apply-config --insecure --mode=interactive --nodes
```
Once the interactive installation is applied, the cluster will form and you can then use `kubectl`.
## Retrieve the `kubeconfig`
Retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl kubeconfig
```
## Upgrading
For example, to upgrade to the latest version of Talos, you can run:
# Orange Pi 5
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/single-board-computers/orangepi_5
Installing Talos on Orange Pi 5 using raw disk image.
## Prerequisites
Before you start:
* follow [Installation/talosctl](../../getting-started/talosctl) to intall `talosctl`
## Boot options
You can boot Talos from:
1. booting from SD card
2. booting from a USB or NVMe (requires a spi image on the SPI flash)
### Booting from SD card
Go to `https://factory.talos.dev` select `Single Board Computers`, select the version and select `Orange Pi 5` from the options.
Choose your desired extensions and fill in the kernel command line arguments if needed.
Download the disk image and decompress it:
#### Flash the image
The image can be flashed using Etcher on Windows, macOS, or Linux or using dd on Linux:
```bash theme={null}
# Replace /dev/ with the destination device
# You can find the device with `lsblk` or `fdisk -l`
sudo dd if=metal-arm64.raw of=/dev/ bs=1M status=progress && sync
```
Proceed by following the [getting started guide](../../getting-started/getting-started#configure-talos-linux) for further steps on how to configure Talos.
#### Booting from USB or NVMe
#### Requirements
* An SD card to boot the Orange Pi 5 board from in order to flash the SPI flash.
Go to `https://factory.talos.dev` select `Single Board Computers`, select the version and select `Orange Pi 5` from the options.
Choose your desired extensions and fill in the kernel command line arguments if needed.
You should also add the `spi_boot: true` overlay extra option in order to remove u-boot from the final image, as the bootloader will be flashed to the SPI flash.
Download the disk image and decompress it:
#### Steps
1. Make sure to install the NVMe or USB drive in the Orange Pi 5 board.
2. Boot the Orange Pi 5 board from the SD card:
* Flash the Orange Pi 5 variant of [Ubuntu](http://www.orangepi.org/html/hardWare/computerAndMicrocontrollers/service-and-support/Orange-pi-5.html) to an SD card.
* Insert the SD card into the Orange Pi 5 board.
* Boot into the Ubuntu image.
* Download [crane CLI](https://github.com/google/go-containerregistry/releases) on the Ubuntu image.
3. From the Ubuntu image, find the latest `sbc-rockchip` overlay, download and extract the u-boot SPI image:
* Find the latest release tag of the [sbc-rockchip repo](https://github.com/siderolabs/sbc-rockchip/releases).
* Download and extract the u-boot SPI image:
```bash theme={null}
crane --platform=linux/arm64 export ghcr.io/siderolabs/sbc-rockchip: | tar x --strip-components=4 artifacts/arm64/u-boot/orangepi-5/u-boot-rockchip-spi.bin
```
4. Flash the SPI flash with the u-boot SPI image:
```bash theme={null}
devicesize=$(blockdev --getsz /dev/mtdblock0)
dd if=/dev/zero of=/dev/mtdblock0 bs=1M count=$devicesize status=progress && sync
dd if=u-boot-rockchip-spi.bin of=/dev/mtdblock0 bs=1M status=progress && sync
```
5. Flash the Talos raw image to the NVMe or USB drive:
```bash theme={null}
sudo dd if=metal-arm64.raw of=/dev/ bs=1M status=progress && sync
```
6. Shutdown the Orange Pi 5 board and remove the SD card.
On the next boot, Talos will now boot from the NVMe/USB and enter maintenance mode.
Proceed by following the [getting started guide](../../getting-started/getting-started#configure-talos-linux) for further steps on how to configure Talos.
## Troubleshooting
### Serial console
If you experience any issues you can check the serial console.
Follow the [official guideline](https://drive.google.com/drive/folders/1ob_qOW2MMa7oncxIW6625NqwXHRxdeAo) (Section 2.18 — "How to use the debugging serial port")
on how to connect a serial adapter.
# Orange Pi R1 Plus LTS
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/single-board-computers/orangepi_r1_plus_lts
Installing Talos on Orange Pi R1 Plus LTS SBC using raw disk image.
## Prerequisites
You will need
* `talosctl`
* an SD card
Download the latest `talosctl`.
## Download the image using Image Factory
The default schematic id for "vanilla" Orange Pi R1 Plus LTS is `da388062cd9318efdc7391982a77ebb2a97ed4fbda68f221354c17839a750509`.
Refer to the [Image Factory](../../learn-more/image-factory) documentation for more information.
Download the image and decompress it:
## Writing the image
The path to your SD card can be found using `fdisk` on Linux or `diskutil` on macOS.
In this example, we will assume `/dev/mmcblk0`.
Now `dd` the image to your SD card:
```bash theme={null}
sudo dd if=metal-arm64.raw of=/dev/mmcblk0 conv=fsync bs=4M
```
## Bootstrapping the Node
Insert the SD card to your board, turn it on and wait for the console to show you the instructions for bootstrapping the node.
Following the instructions in the console output to connect to the interactive installer:
```bash theme={null}
talosctl apply-config --insecure --mode=interactive --nodes
```
Once the interactive installation is applied, the cluster will form and you can then use `kubectl`.
## Retrieve the `kubeconfig`
Retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl kubeconfig
```
## Upgrading
For example, to upgrade to the latest version of Talos, you can run:
# Pine64
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/single-board-computers/pine64
Installing Talos on a Pine64 SBC using raw disk image.
## Prerequisites
You will need
* `talosctl`
* an SD card
Download the latest `talosctl`.
## Download the image
The default schematic id for "vanilla" Pine64 is `185431e0f0bf34c983c6f47f4c6d3703aa2f02cd202ca013216fd71ffc34e175`.
Refer to the [Image Factory](../../learn-more/image-factory) documentation for more information.
Download the image and decompress it:
## Writing the image
The path to your SD card can be found using `fdisk` on Linux or `diskutil` on macOS.
In this example, we will assume `/dev/mmcblk0`.
Now `dd` the image to your SD card:
```bash theme={null}
sudo dd if=metal-arm64.raw of=/dev/mmcblk0 conv=fsync bs=4M
```
## Bootstrapping the node
Insert the SD card to your board, turn it on and wait for the console to show you the instructions for bootstrapping the node.
Following the instructions in the console output to connect to the interactive installer:
```bash theme={null}
talosctl apply-config --insecure --mode=interactive --nodes
```
Once the interactive installation is applied, the cluster will form and you can then use `kubectl`.
## Retrieve the `kubeconfig`
Retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl kubeconfig
```
## Upgrading
For example, to upgrade to the latest version of Talos, you can run:
# Radxa ROCK 4C Plus
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/single-board-computers/rock4cplus
Installing Talos on Radxa ROCK 4c Plus SBC using raw disk image.
## Prerequisites
You will need
* `talosctl`
* an SD card or an eMMC or USB drive or an nVME drive
Download the latest `talosctl`.
## Download the image
The default schematic id for "vanilla" Rock 4c Plus is `ed7091ab924ef1406dadc4623c90f245868f03d262764ddc2c22c8a19eb37c1c`.
Refer to the [Image Factory](../../learn-more/image-factory) documentation for more information.
Download the image and decompress it:
## Writing the Image
The path to your SD card/eMMC/USB/nVME can be found using `fdisk` on Linux or `diskutil` on macOS.
In this example, we will assume `/dev/mmcblk0`.
Now `dd` the image to your SD card:
```bash theme={null}
sudo dd if=metal-arm64.raw of=/dev/mmcblk0 conv=fsync bs=4M
```
The user has two options to proceed:
* booting from a SD card or eMMC
### Booting from SD card or eMMC
Insert the SD card into the board, turn it on and proceed to [bootstrapping the node](#bootstrapping-the-node).
## Bootstrapping the node
Wait for the console to show you the instructions for bootstrapping the node.
Following the instructions in the console output to connect to the interactive installer:
```bash theme={null}
talosctl apply-config --insecure --mode=interactive --nodes
```
Once the interactive installation is applied, the cluster will form and you can then use `kubectl`.
## Retrieve the `kubeconfig`
Retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl kubeconfig
```
## Upgrading
For example, to upgrade to the latest version of Talos, you can run:
# Radxa ROCK 5B
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/single-board-computers/rock5b
Installing Talos on Radxa ROCK 5B SBC using raw disk image.
## Prerequisites
You will need
* follow [Installation/talosctl](../../getting-started/talosctl) to intall `talosctl`
* an SD card
## Download the image
Visit the [Image Factory](https://factory.talos.dev/), select `Single Board Computers`, select the version and select `Radxa ROCK 5B` from the options.
Choose `realtek-firmware` and any other desired extension.
Next fill in the kernel command line arguments if needed.
Download the image and decompress it:
## Writing the image
This guide assumes the node should boot from SD card.
Booting from eMMC or NVMe has not been tested yet.
The path to your SD card can be found using `fdisk` on Linux or `diskutil` on macOS.
In this example, we will assume `/dev/mmcblk0`.
Now `dd` the image to your SD card:
```bash theme={null}
sudo dd if=metal-arm64.raw of=/dev/mmcblk0 conv=fsync oflag=direct status=progress bs=4M
```
## First boot
Insert the SD card into the board, turn it on and proceed by following the
[getting started guide](../../getting-started/getting-started#configure-talos-linux)
for further steps on how to configure Talos.
## Troubleshooting
### Serial console
If you experience any issues you can check the serial console.
Follow the [official guideline](https://wiki.radxa.com/Rock5/dev/serial-console)
on how to connect a serial adapter.
Hint: The rock5b overlay uses baudrate of `115200` instead of the default `1500000`
### Power supplies and endless restarts
It is a known issue that USB Power Delivery negotiation is performed at a late stage in kernel.
This can lead to endless restarts if the power supply cuts power to early.
Check the list of [known working](https://wiki.radxa.com/Rock5/5b/power_supply) power supplies.
## Tips and tricks
### EPHEMERAL on NVMe
The Radxa ROCK 5B SBC provides a M.2 NVMe SSD slot.
This allows to use a separate disk for the EPHEMERAL partition by following
[Disk Management](../../configure-your-talos-cluster/storage-and-disk-management/disk-management).
# Pine64 Rock64
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/single-board-computers/rock64
Installing Talos on Pine64 Rock64 SBC using raw disk image.
## Prerequisites
You will need
* `talosctl`
* an SD card
Download the latest `talosctl`.
## Download the image
The default schematic id for "vanilla" Pine64 Rock64 is `0e162298269125049a51ec0a03c2ef85405a55e1d2ac36a7ef7292358cf3ce5a`.
Refer to the [Image Factory](../../learn-more/image-factory) documentation for more information.
Download the image and decompress it:
## Writing the image
The path to your SD card can be found using `fdisk` on Linux or `diskutil` on macOS.
In this example, we will assume `/dev/mmcblk0`.
Now `dd` the image to your SD card:
```bash theme={null}
sudo dd if=metal-arm64.raw of=/dev/mmcblk0 conv=fsync bs=4M
```
## Bootstrapping the node
Insert the SD card to your board, turn it on and wait for the console to show you the instructions for bootstrapping the node.
Following the instructions in the console output to connect to the interactive installer:
```bash theme={null}
talosctl apply-config --insecure --mode=interactive --nodes
```
Once the interactive installation is applied, the cluster will form and you can then use `kubectl`.
## Retrieve the `kubeconfig`
Retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl kubeconfig
```
## Upgrading
For example, to upgrade to the latest version of Talos, you can run:
# Radxa ROCK PI 4
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/single-board-computers/rockpi_4
Installing Talos on Radxa ROCK PI 4a/4b SBC using raw disk image.
## Prerequisites
You will need
* `talosctl`
* an SD card or an eMMC or USB drive or an nVME drive
Download the latest `talosctl`.
## Download the image
The default schematic id for "vanilla" RockPi 4 is `25d2690bb48685de5939edd6dee83a0e09591311e64ad03c550de00f8a521f51`.
Refer to the [Image Factory](../../learn-more/image-factory) documentation for more information.
Download the image and decompress it:
## Writing the image
The path to your SD card/eMMC/USB/nVME can be found using `fdisk` on Linux or `diskutil` on macOS.
In this example, we will assume `/dev/mmcblk0`.
Now `dd` the image to your SD card:
```bash theme={null}
sudo dd if=metal-arm64.raw of=/dev/mmcblk0 conv=fsync bs=4M
```
The user has two options to proceed:
* booting from a SD card or eMMC
* booting from a USB or nVME (requires the RockPi board to have the SPI flash)
### Booting from SD card or eMMC
Insert the SD card into the board, turn it on and proceed to [bootstrapping the node](#bootstrapping-the-node).
### Booting from USB or nVME
This requires the user to flash the RockPi SPI flash with u-boot.
Follow the Radxa docs on [Install on M.2 NVME SSD](https://wiki.radxa.com/Rockpi4/install/NVME)
After these above steps, Talos will boot from the nVME/USB and enter maintenance mode.
Proceed to [bootstrapping the node](#bootstrapping-the-node).
## Bootstrapping the node
Wait for the console to show you the instructions for bootstrapping the node.
Following the instructions in the console output to connect to the interactive installer:
```bash theme={null}
talosctl apply-config --insecure --mode=interactive --nodes
```
Once the interactive installation is applied, the cluster will form and you can then use `kubectl`.
## Retrieve the `kubeconfig`
Retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl kubeconfig
```
## Upgrading
For example, to upgrade to the latest version of Talos, you can run:
# Radxa ROCK PI 4C
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/single-board-computers/rockpi_4c
Installing Talos on Radxa ROCK PI 4c SBC using raw disk image.
## Prerequisites
You will need
* `talosctl`
* an SD card or an eMMC or USB drive or an nVME drive
Download the latest `talosctl`.
## Download the image
The default schematic id for "vanilla" RockPi 4c is `08e72e242b71f42c9db5bed80e8255b2e0d442a372bc09055b79537d9e3ce191`.
Refer to the [Image Factory](../../learn-more/image-factory) documentation for more information.
Download the image and decompress it:
## Writing the image
The path to your SD card/eMMC/USB/nVME can be found using `fdisk` on Linux or `diskutil` on macOS.
In this example, we will assume `/dev/mmcblk0`.
Now `dd` the image to your SD card:
```bash theme={null}
sudo dd if=metal-arm64.raw of=/dev/mmcblk0 conv=fsync bs=4M
```
The user has two options to proceed:
* booting from a SD card or eMMC
* booting from a USB or nVME (requires the RockPi board to have the SPI flash)
### Booting from SD card or eMMC
Insert the SD card into the board, turn it on and proceed to [bootstrapping the node](#bootstrapping-the-node).
### Booting from USB or nVME
This requires the user to flash the RockPi SPI flash with u-boot.
Follow the Radxa docs on [Install on M.2 NVME SSD](https://wiki.radxa.com/Rockpi4/install/NVME)
After these above steps, Talos will boot from the nVME/USB and enter maintenance mode.
Proceed to [bootstrapping the node](#bootstrapping-the-node).
## Bootstrapping the node
Wait for the console to show you the instructions for bootstrapping the node.
Following the instructions in the console output to connect to the interactive installer:
```bash theme={null}
talosctl apply-config --insecure --mode=interactive --nodes
```
Once the interactive installation is applied, the cluster will form and you can then use `kubectl`.
## Retrieve the `kubeconfig`
Retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl kubeconfig
```
## Upgrading
For example, to upgrade to the latest version of Talos, you can run:
# Raspberry Pi Series
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/single-board-computers/rpi_generic
Installing Talos on Raspberry Pi SBC's using raw disk image.
Talos disk image for the Raspberry Pi generic should in theory work for the boards supported by [u-boot](https://github.com/u-boot/u-boot/blob/master/doc/board/broadcom/raspberrypi.rst#64-bit) `rpi_arm64_defconfig`.
This has only been officialy tested on the Raspberry Pi 4 and community tested on one variant of the Compute Module 4 using Super 6C boards.
If you have tested this on other Raspberry Pi boards, please let us know.
## Video walkthrough
To see a live demo of this writeup, see the video below:
## Prerequisites
You will need
* `talosctl`
* an SD card
Download the latest `talosctl`.
```bash theme={null}
curl -sL 'https://www.talos.dev/install' | bash
```
## Updating the EEPROM
Use [Raspberry Pi Imager](https://www.raspberrypi.com/software/) to write an EEPROM update image to a spare SD card.
Select Misc utility images under the Operating System tab.
Remove the SD card from your local machine and insert it into the Raspberry Pi.
Power the Raspberry Pi on, and wait at least 10 seconds.
If successful, the green LED light will blink rapidly (forever), otherwise an error pattern will be displayed.
If an HDMI display is attached to the port closest to the power/USB-C port,
the screen will display green for success or red if a failure occurs.
Power off the Raspberry Pi and remove the SD card from it.
> Note: Updating the bootloader only needs to be done once.
## Download the image
> Note: if you need to enable Broadcom VideoCore GPU support, generate a new image from the [Image Factory](../../learn-more/image-factory) with the correct [config.txt](#configtxt-information) configuration and `vc4` system extension.
> More information can be found under the [Image Factory Example](#example-raspberry-pi-generic-with-broadcom-videocore-gpu-support-with-image-factory) below.
The default schematic id for "vanilla" Raspberry Pi generic image is `ee21ef4a5ef808a9b7484cc0dda0f25075021691c8c09a276591eedb638ea1f9`.Refer to the [Image Factory](../../learn-more/image-factory) documentation for more information.
Download the image and decompress it:
## Writing the image
Now `dd` the image to your SD card:
```bash theme={null}
sudo dd if=metal-arm64.raw of=/dev/mmcblk0 conv=fsync bs=4M
```
## Bootstrapping the node
Insert the SD card to your board, turn it on and wait for the console to show you the instructions for bootstrapping the node.
Following the instructions in the console output to connect to the interactive installer:
> Note: Add the [vc4 System Extension](https://github.com/siderolabs/extensions/pkgs/container/vc4) for V3D/VC4 Broadcom VideoCore GPU support.
```bash theme={null}
talosctl apply-config --insecure --mode=interactive --nodes
```
Once the interactive installation is applied, the cluster will form and you can then use `kubectl`.
> Note: if you have an HDMI display attached and it shows only a rainbow splash,
> please use the other HDMI port, the one closest to the power/USB-C port.
## Retrieve the `kubeconfig`
Retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl kubeconfig
```
## Upgrading
For example, to upgrade to the latest version of Talos, you can run:
### Example: Raspberry Pi generic with Broadcom VideoCore GPU support with Image Factory
Let's assume we want to boot Talos on a Raspberry Pi with the `vc4` system extension for V3D/VC4 Broadcom VideoCore GPU support.
> Note: The `vc4` system extension requires sufficient Contiguous Memory Allocator (CMA) size for GPU operations.
> See the [GPU Memory Issues](#gpu-memory-issues) section for recommended CMA sizes and configuration steps to prevent errors like `DRM_IOCTL_MODE_CREATE_DUMB failed`.
First, let's create the schematic file `rpi_generic.yaml`:
#### Schematic example with `vc4` system extension
```yaml theme={null}
# rpi_generic.yaml
overlay:
name: rpi_generic
image: siderolabs/sbc-raspberrypi
options:
configTxt: |
gpu_mem=128
kernel=u-boot.bin
arm_64bit=1
arm_boost=1
enable_uart=1
dtoverlay=disable-bt
dtoverlay=disable-wifi
avoid_warnings=2
dtoverlay=vc4-kms-v3d,noaudio
customization:
systemExtensions:
officialExtensions:
- siderolabs/vc4
```
> The schematic doesn't contain any system extension or overlay versions, Image Factory will pick the correct version matching Talos Linux release.
And now we can upload the schematic to the Image Factory to retrieve its ID:
```shell theme={null}
$ curl -X POST --data-binary @rpi_generic.yaml https://factory.talos.dev/schematics
{"id":"0db665edfda21c70194e7ca660955425d16cec2aa58ff031e2abf72b7c328585"}
```
The returned schematic ID `0db665edfda21c70194e7ca660955425d16cec2aa58ff031e2abf72b7c328585` we will use to generate the boot assets.
> The schematic ID is based on the schematic contents, so uploading the same schematic will return the same ID.
Now we can download the metal arm64 image:
* [https://factory.talos.dev/image/0db665edfda21c70194e7ca660955425d16cec2aa58ff031e2abf72b7c328585/\$\{release\_v1\_13}/metal-arm64.raw.xz](https://factory.talos.dev/image/0db665edfda21c70194e7ca660955425d16cec2aa58ff031e2abf72b7c328585/\$\{release_v1_13}/metal-arm64.raw.xz) (download it and burn to a boot media)
> The Image Factory URL contains both schematic ID and Talos version, and both can be changed to generate different boot assets.
Once installed, the machine can be upgraded to a new version of Talos by referencing new installer image:
```shell theme={null}
talosctl upgrade --image factory.talos.dev/installer/0db665edfda21c70194e7ca660955425d16cec2aa58ff031e2abf72b7c328585:
```
Same way upgrade process can be used to transition to a new set of system extensions: generate new schematic with the new set of system extensions, and upgrade the machine to the new schematic ID:
### Example: Raspberry Pi generic with Broadcom VideoCore GPU support with Imager
Let's assume we want to boot Talos on Raspberry Pi with `rpi_generic` overlay and the `vc4` system extension for Broadcom VideoCore GPU support.
> Note: The `vc4` system extension requires sufficient Contiguous Memory Allocator (CMA) size for GPU operations.
> See the [GPU Memory Issues](#gpu-memory-issues) section for recommended CMA sizes and configuration steps to prevent errors like `DRM_IOCTL_MODE_CREATE_DUMB failed`.
First, let's lookup extension images for `vc4` in the [extensions repository](https://github.com/siderolabs/extensions):
Next we'll lookup the overlay image for `rpi_generic` in the [overlays repository](https://github.com/siderolabs/overlays):
Now we can generate the metal image with the following command:
Now the `_out/metal-arm64.raw.xz` is the compressed disk image which can be written to a boot media.
As the next step, we should generate a custom `installer` image which contains all required system extensions (kernel args can't be specified with the installer image, but they are set in the machine configuration):
The `installer` container image should be pushed to the container registry:
Now we can use the customized `installer` image to install Talos on Raspberry Pi.
When it's time to upgrade a machine, a new `installer` image can be generated using the new version of `imager`, and updating the system extension and overlay images to the matching versions.
The custom `installer` image can now be used to upgrade Talos machine.
## config.txt information
Refer to the default [config.txt](https://github.com/siderolabs/sbc-raspberrypi/blob/main/installers/rpi_generic/src/config.txt) file used by the [sbc-raspberrypi](https://github.com/siderolabs/sbc-raspberrypi) overlay.
### Overlay options
The `rpi_generic` overlay provides two options for customizing the `config.txt` boot configuration file:
| Option | Description |
| ----------------- | ---------------------------------------------------------------- |
| `configTxt` | Replaces the entire `config.txt` file with the provided contents |
| `configTxtAppend` | Appends the provided contents to the existing `config.txt` file |
These options must be set at image generation time using the [Image Factory](../../learn-more/image-factory) or Imager.
Changing `config.txt` on a running system is not supported. To apply any changes, you must generate new installation media.
Customizations to `config.txt` are not preserved automatically during upgrades.
If you are using Omni to manage your machines, custom `config.txt` is not currently supported. Omni does not preserve or apply these changes, and upgrading with a custom image is not permitted.
### Configure the `config.txt` file for usage with the `vc4` system extension
> Note: Incorrect GPU memory settings in `config.txt` can lead to rendering failures with the `vc4` extension.
> See the [GPU Memory Issues](#gpu-memory-issues) section for guidance on setting the Contiguous Memory Allocator (CMA) size via `cma-*` parameters.
```ini theme={null}
...
gpu_mem=128 # <== Add or edit this line
...
hdmi_safe:0=1 # <== Remove this line
hdmi_safe:1=1 # <== Remove this line
...
avoid_warnings=2 # <== Add this line
dtoverlay=vc4-kms-v3d,noaudio # <== Add this line
...
```
### Controlling the fan
With an overlay, you are able to controlling the fan using either GPIO or PWM. An example of fan, is the official [Raspberry Pi 4 Case Fan](https://www.raspberrypi.com/products/raspberry-pi-4-case-fan/).
To configure the fan, you either use `gpio-fan` or `pwm-gpio-fan`. Using PWM gives you the option of having four steps of fan speed. Both have extra options that is linked to below.
```ini theme={null}
...
dtoverlay=gpio-fan,gpiopin=14 # <== Defaults to GPIO 12
# or
dtoverlay=pwm-gpio-fan # <== Defaults to GPIO 18
...
```
**NOTE!** GPIO 14 is instructed by the official case fan. But, when using GPIO 14, you must make sure that you either disable UART (`enable_uart=1`), or alternatively change the pin to e.g. GPIO 4 which is one step away. [*Pin layout*](https://pinout.xyz).
#### Documentation
* [`gpio-fan`](https://github.com/raspberrypi/linux/blob/957fc9f9724b3b973a7842d5abbd382b7e3f5e2d/arch/arm/boot/dts/overlays/README#L1524)
* [`pwm-gpio-fan`](https://github.com/raspberrypi/linux/blob/957fc9f9724b3b973a7842d5abbd382b7e3f5e2d/arch/arm/boot/dts/overlays/README#L4189)
## Troubleshooting
The following table can be used to troubleshoot booting issues:
| Long Flashes | Short Flashes | Status |
| ------------ | :-----------: | ----------------------------------: |
| 0 | 3 | Generic failure to boot |
| 0 | 4 | start\*.elf not found |
| 0 | 7 | Kernel image not found |
| 0 | 8 | SDRAM failure |
| 0 | 9 | Insufficient SDRAM |
| 0 | 10 | In HALT state |
| 2 | 1 | Partition not FAT |
| 2 | 2 | Failed to read from partition |
| 2 | 3 | Extended partition not FAT |
| 2 | 4 | File signature/hash mismatch - Pi 4 |
| 4 | 4 | Unsupported board type |
| 4 | 5 | Fatal firmware error |
| 4 | 6 | Power failure type A |
| 4 | 7 | Power failure type B |
### GPU memory issues
The Contiguous Memory Allocator (CMA) reserves physically contiguous memory for Raspberry Pi GPU/display operations (e.g., KMS/DRM rendering).
An error like `DRM_IOCTL_MODE_CREATE_DUMB failed: Cannot allocate memory` indicates an undersized CMA pool for graphics tasks.
In Talos Linux, the Raspberry Pi 4’s CMA size is set via bootloader or kernel parameters.
The default may be too small for GPU-intensive tasks, and oversizing reduces system memory.
**CMA Size Guide**:
| CMA Size | Suitable For |
| :------: | :---------------------------- |
| 64 MB | Headless, no GPU use |
| 128 MB | Light use |
| 256 MB | HD media, cameras |
| 512 MB | 4K media, ML with GPU |
| 1024 MB | Experimental, may destabilize |
#### Change CMA size
**Kernel Parameters**:
Set CMA size with the `cma` parameter (e.g., `cma=256M`, `cma=384M`).
Locate and apply the `machine.kernel` section to the machine config:
```yaml theme={null}
machine:
kernel:
extraArgs:
- cma=256M
```
**Bootloader via Talos Image Factory**:
> Talos’s immutable image requires `config.txt` changes via Talos Image Factory, direct `/boot/config.txt` edits are lost on upgrades.
In Image Factory, Set CMA size with the `cma-*` parameter in `ConfigTxt` (e.g., `cma-128`, `cma-256`):
```ini theme={null}
...
overlay:
...
options:
configTxt: |-
...
dtoverlay=vc4-kms-v3d,cma-256
...
```
# Turing RK1
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/single-board-computers/turing_rk1
Installing Talos on Turing RK1 SOM using raw disk image.
## Prerequisites
Before you start, ensure you have:
* `talosctl`
* `tpi` from [github](https://github.com/turing-machines/tpi/releases)
* [crane CLI](https://github.com/google/go-containerregistry/releases)
Download the latest `talosctl`.
## Download the image
Go to `https://factory.talos.dev` select `Single Board Computers`, select the version and select `Turing RK1` from the options.
Choose your desired extensions and fill in the kernel command line arguments if needed.
Download the disk image and decompress it:
```bash theme={null}
curl -LO https://factory.talos.dev/image/[uuid]/v1.9.0/metal-arm64.raw.xz
xz -d metal-arm64.raw.xz
```
## Boot options
You can boot Talos from:
1. booting from eMMC
2. booting from a USB or NVMe (requires a spi image on the eMMC)
### Booting from eMMC
Flash the image to the eMMC and power on the node: (or use the WebUI of the Turing Pi 2)
```bash theme={null}
tpi flash -n -i metal-arm64.raw
tpi power on -n
```
Proceed to [bootstrapping the node](#bootstrapping-the-node).
### Booting from USB or NVMe
#### Requirements
To boot from USB or NVMe, flash a u-boot SPI image (part of the SBC overlay) to the eMMC.
#### Steps
Skip step 1 if you already installed your NVMe drive.
1. If you have a USB to NVMe adapter, write Talos image to the USB drive:
```bash theme={null}
sudo dd if=metal-arm64.raw of=/dev/sda
```
2. Install the NVMe drive in the Turing Pi 2 board.
If the NVMe drive is/was already installed:
* Flash the Turing RK1 variant of [Ubuntu](https://docs.turingpi.com/docs/turing-rk1-flashing-os) to the eMMC.
* Boot into the Ubuntu image and write the Talos image directly to the NVMe drive:
```bash theme={null}
sudo dd if=metal-arm64.raw of=/dev/nvme0n1
```
3. Find the latest `sbc-rockchip` overlay, download and extract the SBC overlay image:
* Find the latest release tag of the [sbc-rockchip repo](https://github.com/siderolabs/sbc-rockchip/releases).
* Download the sbc overlay image and extract the SPI image:
```bash theme={null}
crane --platform=linux/arm64 export ghcr.io/siderolabs/sbc-rockchip: | tar x --strip-components=4 artifacts/arm64/u-boot/turingrk1/u-boot-rockchip-spi.bin
```
4. Flash the eMMC with the Talos raw image (even if Talos was previously installed): (or use the WebUI of the Turing Pi 2)
```bash theme={null}
tpi flash -n -i metal-turing_rk1-arm64.raw
```
5. Flash the SPI image to set the boot order and remove unnecessary partitions: (or use the WebUI of the Turing Pi 2)
```bash theme={null}
tpi flash -n -i u-boot-rockchip-spi.bin
tpi power on -n
```
Talos will now boot from the NVMe/USB and enter maintenance mode.
## Bootstrapping the node
To monitor boot messages, run: (repeat)
```sh theme={null}
tpi uart -n get
```
Wait until instructions for bootstrapping appear.
Follow the UART instructions to connect to the interactive installer:
```bash theme={null}
talosctl apply-config --insecure --mode=interactive --nodes
```
Alternatively, generate and apply a configuration:
```bash theme={null}
talosctl gen config
talosctl apply-config --insecure --nodes -f .yaml
```
Copy your `talosconfig` to `~/.talos/config` and fill in the `node` field with the IP address of the node and endpoints.
Once applied, the cluster will form, and you can use `kubectl`.
## Retrieve the `kubeconfig`
Retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl kubeconfig
```
# Unofficial Ports
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/single-board-computers/unofficial
List of unofficial ports of Talos Linux to single-board computers.
All single-board computer (SBC) ports are community-maintained and provided as-is.
They are not part of the official Talos Linux source tree and live in the [overlays repository](https://github.com/siderolabs/overlays).
While Talos contributors may offer best-effort help for Talos-specific issues (for example, if Kubernetes doesn’t run properly on a supported architecture), we do not provide official support for SBC-specific problems such as boot failures, firmware quirks, or custom kernel requirements.
If you need official support for SBC boards please contact [sales@siderolabs.com](mailto:sales@siderolabs.com) to discuss a support contract.
## Raspberry Pi 5
* [Talos builder](https://github.com/talos-rpi5/talos-builder)
## MIXTILE Blade 3
* [Installing Talos on MIXTILE Blade 3](https://www.mixtile.com/docs/installing-talos-on-mixtile-blade-3/)
* [Compile Talos Linux Image from Source Code](https://www.mixtile.com/docs/compile-talos-os-image-from-source-code/)
# Hyper-V
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/virtualized-platforms/hyper-v
Creating a Talos Kubernetes cluster using Hyper-V.
## Prerequisites
1. Download the latest `metal-amd64.iso` from the [GitHub releases page](https://github.com/siderolabs/talos/releases).
2. Create a `New-TalosVM` folder in one of your PS Module Path folders (`$env:PSModulePath -split ';'`) and save the [New-TalosVM.psm1](https://github.com/nebula-it/New-TalosVM/blob/main/Talos/1.0.0/Talos.psm1) there.
3. **Create a Virtual Switch**:
* Open Hyper-V Manager (`Win + R`, type `virtmgmt.msc`, press **Enter**).
* From the **Action** menu, select **Virtual Switch Manager...**.
* Choose **New virtual network switch → External**, then click **Create Virtual Switch**.
* Name the switch **LAB**, select the network adapter you want to bind to, and click **Apply** and **OK**.
* Verify that the **LAB** switch appears in the list of virtual switches.
4. **Allow PowerShell Script Execution**:
* Open PowerShell and run:
```powershell theme={null}
Set-ExecutionPolicy Unrestricted -Scope CurrentUser
```
* Confirm the change when prompted.
## Plan overview
We will create a basic 3-node cluster with one control-plane node and two worker nodes.
The main difference between the control plane and worker nodes is the amount of RAM and an additional storage VHD for the worker nodes.
This can be customized to your preference.
We use a `VMNamePrefix` argument for the VM name prefix, not the full hostname.
This command will find any existing VM with that prefix and increment the highest suffix found.
For example, if `talos-cp01` and `talos-cp02` exist, it will create VMs starting from `talos-cp03`, depending on the `NumberOfVMs` argument.
## Setup a control plane node
> Note: Ensure the `LAB` adapter exists in Hyper-V and is set to external.
Create a single control plane node with the following command:
```powershell theme={null}
New-TalosVM -VMNamePrefix talos-cp -CPUCount 2 -StartupMemory 4GB -SwitchName LAB -TalosISOPath C:\ISO\metal-amd64.iso -NumberOfVMs 1 -VMDestinationBasePath 'D:\Virtual Machines\Test VMs\Talos'
```
This will create the `talos-cp01` VM and power it on.
## Setup worker nodes
Create two worker nodes with the following command:
```powershell theme={null}
New-TalosVM -VMNamePrefix talos-worker -CPUCount 4 -StartupMemory 8GB -SwitchName LAB -TalosISOPath C:\ISO\metal-amd64.iso -NumberOfVMs 2 -VMDestinationBasePath 'D:\Virtual Machines\Test VMs\Talos' -StorageVHDSize 50GB
```
This will create `talos-worker01` and `talos-worker02` VMs, each with an additional 50GB VHD for storage (which can be used for Mayastor).
## Push config to the nodes
Once the VMs are ready, find their IP addresses from the VM console.
Push the config to the control plane node with:
```powershell theme={null}
# Set control plane IP variable
$CONTROL_PLANE_IP='10.10.10.x'
# Generate Talos config
talosctl gen config talos-cluster https://$($CONTROL_PLANE_IP):6443 --output-dir .
# Apply config to control plane node
talosctl apply-config --insecure --nodes $CONTROL_PLANE_IP --file .\controlplane.yaml
```
## Push config to worker nodes
Similarly, for the worker nodes:
```powershell theme={null}
talosctl apply-config --insecure --nodes 10.10.10.x --file .\worker.yaml
```
Apply the config to both worker nodes.
## Bootstrap cluster
With the nodes ready, bootstrap the Kubernetes cluster:
```powershell theme={null}
# Set node and endpoint permanently in config
talosctl config endpoint $CONTROL_PLANE_IP
talosctl config node $CONTROL_PLANE_IP
# Bootstrap cluster
talosctl bootstrap
# Generate kubeconfig
talosctl kubeconfig .
```
## Remove ISO
After a successful bootstrap, remove the ISO from the Hyper-V instances (both worker and control plane).
Otherwise, Talos might fail to boot.
This will generate the `kubeconfig` file, which you can use to connect to the cluster.
# KVM
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/virtualized-platforms/kvm
Create a Talos Kubernetes cluster with KVM.
In this guide, you’ll create a Kubernetes cluster using `KVM` and `virsh` command-line utility.
## Requirements
Make sure you have the following installed and configured:
* **Kernel with KVM enabled** (`/dev/kvm` must exist)
* **QEMU**
* **virsh**: Command-line interface for managing KVM/QEMU VMs
* **kubectl**: Kubernetes CLI
* **talosctl**: CLI for interacting with Talos clusters
* **16 GB RAM**: Recommended minimum for running the VMs
* *(Optional)* `virt-manager` for a graphical interface
> **Note:** This guide assumes you are running all the commands from the following working directory.
Create a working directory for your project files:
```bash theme={null}
mkdir -p ~/talos-kvm/configs
cd ~/talos-kvm
```
Download the latest `metal-amd64.iso` from the Talos [GitHub releases page](https://github.com/siderolabs/talos/releases).
## Configure the network
Before we get started, let’s set up an isolated network for your Talos cluster.
Use the following command to write the required network configuration into a file.
```bash theme={null}
cat > my-talos-net.xml <
my-talos-net
EOF
```
```bash theme={null}
cat > my-talos-net.xml <
my-talos-net
EOF
```
Use the following command to generate the configurations and define the bridge:
```bash theme={null}
virsh net-define my-talos-net.xml
```
Use the following command to start the network and flag it to auto start after reboot:
```bash theme={null}
virsh net-start my-talos-net
virsh net-autostart my-talos-net
```
Verify the network:
```bash theme={null}
virsh net-info my-talos-net
```
> **Note:** You should see the network as `Active`.
Expected output:
```bash theme={null}
Name: my-talos-net
UUID:
Active: yes
Persistent: yes
Autostart: yes
Bridge: talos-bridge
```
## Provisioning the environment
Now that you have a dedicated network let's go ahead and provision VMs.
> **Note:** For the network interface emulation, `virtio` and `e1000` are supported.
>
> In the next command, you will use the `--network network=my-talos-net` flag to attach the VMs to the Talos network you created earlier.
>
> The values for --ram and --vcpus in this guide are suggestions. You can adjust these to match the resources available on your host machine and the needs of your workload.
Use the following command to create a controlplane node:
```bash theme={null}
virt-install \
--virt-type kvm \
--name control-plane-node-1 \
--ram 2048 \
--vcpus 2 \
--disk path=control-plane-node-1-disk.qcow2,bus=virtio,size=40,format=qcow2 \
--cdrom metal-amd64.iso \
--os-variant=linux2022 \
--network network=my-talos-net \
--boot hd,cdrom --noautoconsole
```
Use the following command to create a worker node:
```bash theme={null}
virt-install \
--virt-type kvm \
--name worker-node-1 \
--ram 4086 \
--vcpus 2 \
--disk path=worker-node-1-disk.qcow2,bus=virtio,size=40,format=qcow2 \
--cdrom metal-amd64.iso \
--os-variant=linux2022 \
--network network=my-talos-net \
--boot hd,cdrom --noautoconsole
```
Use the following command to verify that your VMs are in a running state:
```bash theme={null}
virsh list
```
## Configure the cluster
Now that you have your VMs provisioned it's time to configure the cluster. This step is done through `talosctl` command utility.
Use the following command to view your control plane IP address, and do the same for your worker node by adjusting the vm name:
```bash theme={null}
virsh domifaddr control-plane-node-1
```
This guide is designed to help you get up and running quickly. To simplify the process, you will store the VM IP addresses in two environment variables: `CP_IP` and `NODE_IP`.
You can automate this step with the following commands:
```bash theme={null}
export CP_IP=$(virsh domifaddr control-plane-node-1 | egrep '/' | awk '{print $4}' | cut -d/ -f1)
export NODE_IP=$(virsh domifaddr worker-node-1 | egrep '/' | awk '{print $4}' | cut -d/ -f1)
```
Currently, your VMs are running Talos directly from the ISO image you downloaded. To install Talos onto the VM disks (so they can boot without the ISO), you need to specify the disk name during the configuration step.
Use the following command to get the name of your VMs disk:
```bash theme={null}
talosctl get disks --nodes $CP_IP --insecure
```
You should see a result similar to the following:
```bash theme={null}
NODE NAMESPACE TYPE ID VERSION SIZE READ ONLY TRANSPORT ROTATIONAL WWID MODEL SERIAL
runtime Disk loop0 2 73 MB true
runtime Disk sr0 2 301 MB false ata QEMU DVD-ROM QEMU_DVD-ROM_QM00001
runtime Disk vda 2 43 GB false virtio true
```
Use the following command to generate the Talos configurations:
```bash theme={null}
talosctl gen config my-talos-cluster https://$CP_IP:6443 --install-disk /dev/vda -o configs/
```
Use the following command to apply the configurations to each VM:
```bash theme={null}
talosctl apply-config --insecure --nodes $CP_IP --file configs/controlplane.yaml
talosctl apply-config --insecure --nodes $NODE_IP --file configs/worker.yaml
```
At this point your VMs will reboot.
## Bootstrapping the cluster
After your VMs restart, you can bootstrap the cluster. Bootstrap simply means starting up your Kubernetes cluster for the first time.
To do this, provide the `talosconfig` file (generated earlier) to the `talosctl` command-line utility. The easiest way is to export the path to `talosconfig` as an environment variable.
Use the following command to export your talosconfig:
```bash theme={null}
export TALOSCONFIG=$(realpath configs/talosconfig)
```
Before running the bootstrap, you need to tell `talosctl` where to connect:
```bash theme={null}
talosctl config endpoint $CP_IP
```
> **Note:** Keep in mind that it may take a couple of seconds for the bootstraping to complete.
Use the following command to bootstrap the cluster:
```bash theme={null}
talosctl -n $CP_IP bootstrap
```
At this point you should be able to see all the VMs that are participating in your cluster.
```bash theme={null}
talosctl -n $CP_IP get members
```
## Accessing the cluster
To access your cluster you need to export the kubeconfig file from the Talos cluster.
Use the following command to export the kubeconfig file:
```bash theme={null}
talosctl -n $CP_IP kubeconfig $PWD/configs/kubeconfig
```
Set the `KUBECONFIG` environment variable to use your new `kubeconfig` file.
```bash theme={null}
export KUBECONFIG=$PWD/configs/kubeconfig
```
That's it! You can now use `kubectl` to interact with your Talos Kubernetes cluster. Check the status of your nodes with `kubectl` to verify they are ready.
```bash theme={null}
kubectl get nodes -o wide
```
## Clean up
When you are finished with your cluster, you can delete the VMs and the isolated network to free up system resources.
```bash theme={null}
virsh destroy
virsh undefine --remove-all-storage
```
Use the following command to delete the network:
```bash theme={null}
virsh net-destroy --network my-talos-net
virsh net-undefine --network my-talos-net
```
# OpenNebula
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/virtualized-platforms/opennebula
Creating a Talos Kubernetes cluster on OpenNebula.
In this guide you will create a Kubernetes cluster on [OpenNebula](https://opennebula.io/).
## Overview
Talos boots into **maintenance mode** on first start, waiting for a machine configuration to be pushed via the Talos API.
OpenNebula provides network configuration to the VM through context variables in `context.sh`.
Talos reads these variables to configure networking before entering maintenance mode, so `talosctl apply-config` can reach the node.
## Prerequisites
* An OpenNebula cluster with at least one hypervisor node
* The OpenNebula CLI tools (`onevm`, `onetemplate`, etc.) or access to the Sunstone web UI
* `talosctl` installed locally ([installation guide](../../getting-started/talosctl))
* `kubectl` installed locally
## Download the Talos disk image
Talos provides pre-built OpenNebula disk images via [Image Factory](https://factory.talos.dev/).
Use the following command to download the disk image:
Use the following command to upload the image to OpenNebula:
## Configure network context
OpenNebula passes network configuration to VMs via context variables in `context.sh`.
Talos reads the `ETH0_*` (and `ETH1_*`, etc.) variables to configure each network interface at boot time.
Talos triggers interface configuration on `ETH*_MAC` key presence, regardless of the `NETWORK` variable.
### Using automatic network context (FIXED and RANGED pools)
Set `NETWORK` to `"YES"` in the VM context so that OpenNebula automatically populates the `ETH*_` variables from the NIC definitions.
This requires the address pool to carry IP data (i.e. `FIXED` or `RANGED` type, not `ETHER`).
```bash theme={null}
CONTEXT = [
NETWORK = "YES"
]
```
### Using manual network variables for MAC-only pools
If the address pool type is `ETHER` (MAC-only, no IP data), set `NETWORK` to `"NO"` and specify the interface parameters manually.
This prevents OpenNebula from overwriting the `ETH*_` variables with empty strings.
```bash theme={null}
CONTEXT = [
NETWORK = "NO",
ETH0_MAC = "$NIC[MAC]",
ETH0_IP = "YOUR_STATIC_IP",
ETH0_GATEWAY = "YOUR_GATEWAY_IP",
ETH0_DNS = "YOUR_DNS_IP"
]
```
Replace `YOUR_STATIC_IP`, `YOUR_GATEWAY_IP`, and `YOUR_DNS_IP` with values appropriate for your network.
The `$NIC[MAC]` expression is resolved by OpenNebula at instantiation time from the NIC definition.
## Create a virtual machine template
Below is a minimal VM template that boots Talos in maintenance mode with a static IP.
Adjust resource values, disk size, and network names for your environment.
Replace `YOUR_NETWORK_NAME` with the name of your OpenNebula virtual network, `YOUR_STATIC_IP` with the desired IP address for the node, and `TALOS_IMAGE_NAME` with the image name used in the upload step above.
```bash theme={null}
onetemplate create --name talos-node << 'EOF'
NAME = "talos-node"
CPU = "2"
VCPU = "2"
MEMORY = "4096"
DISK = [
IMAGE = "TALOS_IMAGE_NAME",
SIZE = "20480"
]
NIC = [
NETWORK = "YOUR_NETWORK_NAME",
IP = "YOUR_STATIC_IP"
]
CONTEXT = [
NETWORK = "YES"
]
GRAPHICS = [
LISTEN = "0.0.0.0",
TYPE = "VNC"
]
OS = [
BOOT = "disk0"
]
EOF
```
## Boot the VMs
Use the following command to start a control plane VM:
```bash theme={null}
onevm create --name talos-cp-1 talos-node
```
Use the following command to start a worker VM:
```bash theme={null}
onevm create --name talos-worker-1 talos-node
```
Use the following command to check the VM state:
```bash theme={null}
onevm list
```
Wait for each VM to reach the `RUNNING` state.
Talos will boot and enter maintenance mode.
You can observe the boot progress from the VNC console in Sunstone, or via `onevm show talos-cp-1`.
## Apply machine configuration
Export the node IP addresses as environment variables:
```bash theme={null}
export CONTROL_PLANE_IP=
export WORKER_IP=
```
Use the following command to generate machine configurations:
```bash theme={null}
talosctl gen config talos-opennebula-cluster https://$CONTROL_PLANE_IP:6443 \
--output-dir _out
```
This creates `_out/controlplane.yaml`, `_out/worker.yaml`, and `_out/talosconfig`.
> **Note:** Check the install disk name before applying the configuration.
> Use `talosctl get disks --insecure --nodes $CONTROL_PLANE_IP` and update `install.disk` in the generated YAML if needed (e.g., `/dev/vda`).
Use the following command to apply the configuration to the control plane node:
```bash theme={null}
talosctl apply-config --insecure --nodes $CONTROL_PLANE_IP --file _out/controlplane.yaml
```
Use the following command to apply the configuration to the worker node:
```bash theme={null}
talosctl apply-config --insecure --nodes $WORKER_IP --file _out/worker.yaml
```
After applying, each node installs Talos to disk and reboots into the configured state.
## Bootstrap the cluster
Use the following commands to configure `talosctl` to use your new cluster:
```bash theme={null}
export TALOSCONFIG="_out/talosconfig"
talosctl config endpoint $CONTROL_PLANE_IP
talosctl config node $CONTROL_PLANE_IP
```
Use the following command to bootstrap etcd on the control plane node:
```bash theme={null}
talosctl bootstrap
```
Use the following command to wait for the control plane to become healthy:
```bash theme={null}
talosctl health
```
## Retrieve kubeconfig
Use the following commands to retrieve the kubeconfig and verify the cluster:
```bash theme={null}
talosctl kubeconfig _out/kubeconfig
export KUBECONFIG=_out/kubeconfig
kubectl get nodes -o wide
```
## Embed machine config using USER\_DATA
Instead of pushing config via the Talos API after boot, you can embed the machine configuration directly in the VM context using the `USER_DATA` variable.
Talos reads `USER_DATA` from the context and applies it automatically on first boot, bypassing maintenance mode.
This method works with any address pool type, including `ETHER`.
```bash theme={null}
CONTEXT = [
USER_DATA = "",
USER_DATA_ENCODING = "base64"
]
```
Use the following commands to generate and encode a machine config:
```bash theme={null}
talosctl gen config talos-opennebula-cluster https://$CONTROL_PLANE_IP:6443 --output-dir _out
base64 -w0 _out/controlplane.yaml
```
Paste the output as the `USER_DATA` value.
> **Security note:** The `USER_DATA` variable is stored in the OpenNebula database and visible via the OpenNebula API to any user with access to the VM template or instance.
> Machine configurations contain sensitive data including cluster CA keys and bootstrap tokens.
> Using `talosctl apply-config` (the default approach above) avoids storing secrets in OpenNebula context entirely.
## Troubleshooting
### Node does not reach maintenance mode
* Verify the context variables injected by OpenNebula using the CLI:
```bash theme={null}
onevm show
```
Check the `CONTEXT` section in the output and confirm `ETH0_MAC`, `ETH0_IP`, and `ETH0_GATEWAY` are present and non-empty.
* If using `NETWORK` set to `"YES"` with an `ETHER`-type pool, OpenNebula sets `ETH0_IP` to an empty string, causing Talos to fail with a parse error.
Switch to `NETWORK` set to `"NO"` with manual `ETH*_` variables as described in the [configure network context section](#configure-network-context),
or use the [USER\_DATA method](#embed-machine-config-using-user_data).
### talosctl apply-config times out
* Confirm the node IP is reachable from your workstation.
* Check that the Talos maintenance mode API port (TCP 50000) is not blocked by a firewall.
* Verify the IP in the context matches what you expect by running `onevm show `.
### Disk not found during install
Use the following command to list available disks while the node is in maintenance mode:
```bash theme={null}
talosctl get disks --insecure --nodes $CONTROL_PLANE_IP
```
Update `install.disk` in your `controlplane.yaml` (or `worker.yaml`) to match the correct device path, then re-apply the configuration.
# Proxmox
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/virtualized-platforms/proxmox
Creating Talos Kubernetes cluster using Proxmox.
In this guide we will create a Kubernetes cluster using Proxmox.
## Video walkthrough
To see a live demo of this writeup, visit Youtube here:
## Installation
### How to get Proxmox
It is assumed that you have already installed Proxmox onto the server you wish to create Talos VMs on.
Visit the [Proxmox](https://www.proxmox.com/en/downloads) downloads page if necessary.
### Install talosctl
You can download `talosctl` on MacOS and Linux via:
```bash theme={null}
brew install siderolabs/tap/talosctl
```
For manual installation and other platforms please see the [talosctl installation guide](../../getting-started/talosctl).
### Download ISO image
In order to install Talos in Proxmox, you will need the ISO image from [Image Factory](https://www.talos.dev/latest/talos-guides/install/boot-assets/#image-factory).
```bash theme={null}
mkdir -p _out/
curl https://factory.talos.dev/image/376567988ad370138ad8b2698212367b8edcb69b5fd68c80be1f2ec7d603b4ba//metal-.iso -L -o _out/metal-.iso
```
For example version for `linux` platform:
### QEMU guest agent support (iso)
* If you need the QEMU guest agent so you can do guest VM shutdowns of your Talos VMs, then you will need a custom ISO
* To get this, navigate to [https://factory.talos.dev/](https://factory.talos.dev/)
* Scroll down and select your Talos version ( for example)
* Then tick the box for `siderolabs/qemu-guest-agent` and submit
* This will provide you with a link to the bare metal ISO
* The lines we're interested in are as follows
- Download the above ISO (this will most likely be `amd64` for you)
- Take note of the `factory.talos.dev/installer` URL as you'll need it later
## Upload ISO
From the Proxmox UI, select the "local" storage and enter the "Content" section.
Click the "Upload" button:
![]()
Select the ISO you downloaded previously, then hit "Upload"
![]()
## Create VMs
Before starting, familiarise yourself with the
[system requirements](../../getting-started/system-requirements) for Talos and assign VM
resources accordingly.
### Recommended baseline VM configuration
Use the following baseline settings for Proxmox VMs running Talos:
| Setting | Recommended Value | Notes |
| ------------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| **BIOS** | `ovmf` (UEFI) | Modern firmware, Secure Boot support, better hardware compatibility |
| **Machine** | `q35` | Modern PCIe-based machine type with better device support |
| **CPU Type** | `host` | Enables advanced instruction sets (AVX-512, etc.), best performance. Alternative: `kvm64` with feature flags for Proxmox \< 8.0 |
| **CPU Cores** | 2+ (control plane), 4+ (workers) | Minimum 2 cores required |
| **Memory** | 4GB+ (control plane), 8GB+ (workers) | Minimum 2GB required |
| **Disk Controller** | **VirtIO SCSI** (NOT "VirtIO SCSI Single") | Single controller can cause bootstrap hangs (#11173) |
| **Disk Format** | Raw (performance) or QCOW2 (features/snapshots) | Raw preferred for performance |
| **Disk Cache** | Write Through (safe default) | Or None for clustered environments |
| **Network Model** | `virtio` | Paravirtualized driver, best performance (up to 10 Gbit) |
| **EFI Disk** | 4MB (for OVMF) | Required for UEFI firmware, stores Secure Boot keys |
| **Ballooning** | Disabled | Talos doesn't support memory hotplug |
| **RNG Device** | VirtIO RNG (optional) | Better entropy for security |
> **Important**: When configuring the disk controller, use **VirtIO SCSI** (not "VirtIO SCSI Single").
> Using "VirtIO SCSI Single" can cause bootstrap to hang or prevent disk discovery.
> See [issue #11173](https://github.com/siderolabs/talos/issues/11173) for details.
Create a new VM by clicking the "Create VM" button in the Proxmox UI:
![]()
Fill out a name for the new VM:
![]()
In the OS tab, select the ISO we uploaded earlier:
![]()
In the "System" tab:
* Set **BIOS** to `ovmf` (UEFI) for modern firmware and Secure Boot support
* Set **Machine** to `q35` for modern PCIe-based machine type
* Add **EFI Disk** (4MB) for persistent UEFI settings and Secure Boot key storage
In the "Hard Disk" tab:
* Set **Bus/Device** to `VirtIO SCSI` (NOT "VirtIO SCSI Single")
* Set **Storage** to your main storage pool
* Set **Format** to `Raw` (better performance) or `QCOW2` (features/snapshots)
* Set **Size** based on your workload requirements (adjust based on CSI and application needs)
* Set **Cache** to `Write Through` (safe default) or `None` for clustered environments
* Enable **Discard** (TRIM support) if using SSD storage
* Enable **SSD emulation** if using SSD storage
> **Important**: When configuring the disk controller, use **VirtIO SCSI** (not "VirtIO SCSI Single").
> Using "VirtIO SCSI Single" can cause bootstrap to hang or prevent disk discovery.
> See [issue #11173](https://github.com/siderolabs/talos/issues/11173) for details.
In the "CPU" section:
* Set **Cores** to 2+ for control planes, 4+ for workers
* Set **Sockets** to 1 (keep simple)
* Set **Type** to `host` (best performance, enables advanced instruction sets)
* **Alternative for Proxmox \< 8.0**: Use `kvm64` with feature flags by adding to `/etc/pve/qemu-server/.conf`:
```text theme={null}
args: -cpu kvm64,+cx16,+lahf_lm,+popcnt,+sse3,+ssse3,+sse4.1,+sse4.2
```
* **Note**: `host` CPU type prevents live VM migration but provides best performance
![]()
In the "Memory" section:
* Set **Memory** to 4GB+ for control planes, 8GB+ for workers (minimum 2GB required)
* **Disable Ballooning** (can cause issues with Talos memory detection)
![]()
In the "Network" section:
* Set **Model** to `virtio` (paravirtualized driver, best performance)
* Set **Bridge** to your network bridge (e.g., `vmbr0`)
* Verify the VM is set to come up on the bridge interface
![]()
> **Tip**: Enable a serial console (ttyS0) in Proxmox VM settings to see early boot logs and troubleshoot network connectivity issues.
> This is especially helpful when debugging DHCP timing or bridge configuration problems.
> Set **Serial port** to `ttyS0` in Proxmox and add `console=ttyS0` if you're customizing kernel args.
Finish creating the VM by clicking through the "Confirm" tab and then "Finish".
Repeat this process for a second VM to use as a worker node.
You can also repeat this for additional nodes desired.
> Note: Talos doesn't support memory hot plugging, if creating the VM programmatically don't enable memory hotplug on your
> Talos VM's.
> Doing so will cause Talos to be unable to see all available memory and have insufficient memory to complete
> installation of the cluster.
## Start control plane node
Once the VMs have been created and updated, start the VM that will be the first control plane node.
This VM will boot the ISO image specified earlier and enter "maintenance mode".
### With DHCP server
Once the machine has entered maintenance mode, there will be a console log that details the IP address that the node received.
Take note of this IP address, which will be referred to as `$CONTROL_PLANE_IP` for the rest of this guide.
If you wish to export this IP as a bash variable, simply issue a command like `export CONTROL_PLANE_IP=1.2.3.4`.
![]()
### Without DHCP server
To apply the machine configurations in maintenance mode, VM has to have IP on the network.
So you can set it on boot time manually.
![]()
Press `e` on the boot time.
And set the IP parameters for the VM.
[Format is](https://www.kernel.org/doc/Documentation/filesystems/nfs/nfsroot.txt):
```bash theme={null}
ip=::::::
```
For example \$CONTROL\_PLANE\_IP will be 192.168.0.100 and gateway 192.168.0.1
```bash theme={null}
linux /boot/vmlinuz init_on_alloc=1 slab_nomerge pti=on panic=0 consoleblank=0 printk.devkmsg=on earlyprintk=ttyS0 console=tty0 console=ttyS0 talos.platform=metal ip=192.168.0.100::192.168.0.1:255.255.255.0::eth0:off
```
![]()
Then press Ctrl-x or F10
## Generate machine configurations
With the IP address above, you can now generate the machine configurations to use for installing Talos and Kubernetes.
Issue the following command, updating the output directory, cluster name, and control plane IP as you see fit:
```bash theme={null}
talosctl gen config talos-proxmox-cluster https://$CONTROL_PLANE_IP:6443 --output-dir _out
```
This will create several files in the `_out` directory: `controlplane.yaml`, `worker.yaml`, and `talosconfig`.
> Note: The Talos config by default will install to `/dev/sda`.
> Depending on your setup the virtual disk may be mounted differently Eg: `/dev/vda`.
> You can check for disks running the following command:
>
> ```bash theme={null}
> talosctl get disks --insecure --nodes $CONTROL_PLANE_IP
> ```
>
> Update `controlplane.yaml` and `worker.yaml` config files to point to the correct disk location.
### QEMU guest agent support
For QEMU guest agent support, you can generate the config with the custom install image:
> **Important**: Enable QEMU Guest Agent in Proxmox **only if** you built the ISO with the `siderolabs/qemu-guest-agent` extension in **Image Factory**.
> If you're using a standard Talos ISO without this extension, leave QEMU Guest Agent disabled in Proxmox VM settings.
> Enabling it without the extension will only generate log spam and won't provide any functionality.
> See: [Image Factory](../../learn-more/image-factory) for building a custom ISO with extensions.
* If you did include the extension, go to your VM → **Options** and set **QEMU Guest Agent** to **Enabled**.
## Create control plane node
Using the `controlplane.yaml` generated above, you can now apply this config using talosctl.
Issue:
```bash theme={null}
talosctl apply-config --insecure --nodes $CONTROL_PLANE_IP --file _out/controlplane.yaml
```
You should now see some action in the Proxmox console for this VM.
Talos will be installed to disk, the VM will reboot, and then Talos will configure the Kubernetes control plane on this VM.
The VM will remain in stage `Booting` until the bootstrap is completed in a later step.
> Note: This process can be repeated multiple times to create an HA control plane.
## Create worker node
Create at least a single worker node using a process similar to the control plane creation above.
Start the worker node VM and wait for it to enter "maintenance mode".
Take note of the worker node's IP address, which will be referred to as `$WORKER_IP`
Issue:
```bash theme={null}
talosctl apply-config --insecure --nodes $WORKER_IP --file _out/worker.yaml
```
> Note: This process can be repeated multiple times to add additional workers.
## Using the cluster
Once the cluster is available, you can make use of `talosctl` and `kubectl` to interact with the cluster.
For example, to view current running containers, run `talosctl containers` for a list of containers in the `system` namespace, or `talosctl containers -k` for the `k8s.io` namespace.
To view the logs of a container, use `talosctl logs ` or `talosctl logs -k `.
First, configure talosctl to talk to your control plane node by issuing the following, updating paths and IPs as necessary:
```bash theme={null}
export TALOSCONFIG="_out/talosconfig"
talosctl config endpoint $CONTROL_PLANE_IP
talosctl config node $CONTROL_PLANE_IP
```
### Bootstrap etcd
```bash theme={null}
talosctl bootstrap
```
### Retrieve the `kubeconfig`
At this point we can retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl kubeconfig .
```
## Troubleshooting
### Cluster creation issues
If `talosctl cluster create` fails with disk controller errors:
* **"virtio-scsi-single disk controller is not supported"**: This disk controller type causes Talos bootstrap to hang. Use `virtio` or `scsi` instead:
```bash theme={null}
# Wrong - will be rejected
talosctl cluster create --disks virtio-scsi-single:10GiB
# Correct - use virtio or scsi
talosctl cluster create --disks virtio:10GiB
talosctl cluster create --disks scsi:10GiB
```
### Network connectivity issues
If nodes fail to obtain IP addresses or show "network is unreachable" errors:
1. **Verify bridge interface**: Ensure the bridge interface (e.g., `vmbr0`) exists and is UP before starting VMs
```bash theme={null}
ip link show vmbr0
```
2. **Check DHCP server**: Ensure DHCP server is running and reachable from the bridge network
3. **Firewall rules**: If Proxmox VM firewall is enabled, allow DHCP traffic (UDP ports 67/68).
If you enforce further filtering, ensure control-plane/API connectivity per your environment's policy (see Talos networking docs).
4. **VLAN configuration**: Ensure VLAN tags match between bridge configuration, VM network settings, and switch configuration
5. **Serial console**: Enable serial console to view early boot logs and network initialization messages
### Disk Controller Issues
* **Configuration rejected**: If you see "virtio-scsi-single disk controller is not supported", use `--disks virtio:10GiB` instead of `--disks virtio-scsi-single:10GiB`
* **Bootstrap hangs**: If bootstrap hangs or disks aren't discovered, verify you're using **VirtIO SCSI** (not "VirtIO SCSI Single")
* **Disk not found**: Check disk path using `talosctl get disks --insecure --nodes $CONTROL_PLANE_IP` and update `install.disk` in machine config if needed (e.g., `install.disk: /dev/vda`)
### Secure boot
For Secure Boot setup, see the [Secure Boot documentation](../bare-metal-platforms/secureboot).
## Cleaning up
To cleanup, simply stop and delete the virtual machines from the Proxmox UI.
# Vagrant & Libvirt
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/virtualized-platforms/vagrant-libvirt
Create a highly available Talos cluster locally using Vagrant and libvirt.
## Pre-requisities
1. Linux OS
2. [Vagrant](https://www.vagrantup.com) installed
3. [vagrant-libvirt](https://github.com/vagrant-libvirt/vagrant-libvirt) plugin installed
4. [talosctl](../../getting-started/talosctl#talosctl) installed
5. [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) installed
## Overview
We will use Vagrant and its libvirt plugin to create a KVM-based cluster with 3 control plane nodes and 1 worker node.
For this, we will mount Talos ISO into the VMs using a virtual CD-ROM,
and configure the VMs to attempt to boot from the disk first with the fallback to the CD-ROM.
We will also configure a virtual IP address on Talos to achieve high-availability on kube-apiserver.
## Preparing the environment
First, we download the latest `metal-amd64.iso` ISO from GitHub releases into the `/tmp` directory.
Create a `Vagrantfile` with the following contents:
```ruby theme={null}
Vagrant.configure("2") do |config|
config.vm.define "control-plane-node-1" do |vm|
vm.vm.provider :libvirt do |domain|
domain.cpus = 2
domain.memory = 2048
domain.serial :type => "file", :source => {:path => "/tmp/control-plane-node-1.log"}
domain.storage :file, :device => :cdrom, :path => "/tmp/metal-amd64.iso"
domain.storage :file, :size => '4G', :type => 'raw'
domain.boot 'hd'
domain.boot 'cdrom'
end
end
config.vm.define "control-plane-node-2" do |vm|
vm.vm.provider :libvirt do |domain|
domain.cpus = 2
domain.memory = 2048
domain.serial :type => "file", :source => {:path => "/tmp/control-plane-node-2.log"}
domain.storage :file, :device => :cdrom, :path => "/tmp/metal-amd64.iso"
domain.storage :file, :size => '4G', :type => 'raw'
domain.boot 'hd'
domain.boot 'cdrom'
end
end
config.vm.define "control-plane-node-3" do |vm|
vm.vm.provider :libvirt do |domain|
domain.cpus = 2
domain.memory = 2048
domain.serial :type => "file", :source => {:path => "/tmp/control-plane-node-3.log"}
domain.storage :file, :device => :cdrom, :path => "/tmp/metal-amd64.iso"
domain.storage :file, :size => '4G', :type => 'raw'
domain.boot 'hd'
domain.boot 'cdrom'
end
end
config.vm.define "worker-node-1" do |vm|
vm.vm.provider :libvirt do |domain|
domain.cpus = 1
domain.memory = 1024
domain.serial :type => "file", :source => {:path => "/tmp/worker-node-1.log"}
domain.storage :file, :device => :cdrom, :path => "/tmp/metal-amd64.iso"
domain.storage :file, :size => '4G', :type => 'raw'
domain.boot 'hd'
domain.boot 'cdrom'
end
end
end
```
## Bring up the nodes
Check the status of vagrant VMs:
```bash theme={null}
vagrant status
```
You should see the VMs in "not created" state:
```text theme={null}
Current machine states:
control-plane-node-1 not created (libvirt)
control-plane-node-2 not created (libvirt)
control-plane-node-3 not created (libvirt)
worker-node-1 not created (libvirt)
```
Bring up the vagrant environment:
```bash theme={null}
vagrant up --provider=libvirt
```
Check the status again:
```bash theme={null}
vagrant status
```
Now you should see the VMs in "running" state:
```text theme={null}
Current machine states:
control-plane-node-1 running (libvirt)
control-plane-node-2 running (libvirt)
control-plane-node-3 running (libvirt)
worker-node-1 running (libvirt)
```
Find out the IP addresses assigned by the libvirt DHCP by running:
```bash theme={null}
virsh -q list --state-running --name | xargs -r -n1 virsh domifaddr
```
Output will look like the following:
```text theme={null}
virsh domifaddr vagrant_control-plane-node-2
Name MAC address Protocol Address
-------------------------------------------------------------------------------
vnet0 52:54:00:f9:10:e5 ipv4 192.168.121.119/24
virsh domifaddr vagrant_control-plane-node-1
Name MAC address Protocol Address
-------------------------------------------------------------------------------
vnet1 52:54:00:0f:ae:59 ipv4 192.168.121.203/24
virsh domifaddr vagrant_worker-node-1
Name MAC address Protocol Address
-------------------------------------------------------------------------------
vnet2 52:54:00:6f:28:95 ipv4 192.168.121.69/24
virsh domifaddr vagrant_control-plane-node-3
Name MAC address Protocol Address
-------------------------------------------------------------------------------
vnet3 52:54:00:03:45:10 ipv4 192.168.121.125/24
```
Our control plane nodes have the IPs: `192.168.121.203`, `192.168.121.119`, `192.168.121.125` and the worker node has the IP `192.168.121.69`.
Now you should be able to interact with Talos nodes that are in maintenance mode:
```bash theme={null}
talosctl -n 192.168.121.203 get disks --insecure
```
Sample output:
```text theme={null}
DEV MODEL SERIAL TYPE UUID WWID MODALIAS NAME SIZE BUS_PATH
/dev/vda - - HDD - - virtio:d00000002v00001AF4 - 8.6 GB /pci0000:00/0000:00:03.0/virtio0/
```
## Installing Talos
Pick an endpoint IP in the `vagrant-libvirt` subnet but not used by any nodes, for example `192.168.121.100`.
Create a patch.yaml file with the following contents and add the virtual IP you picked to a network interface under `.machine.network.interfaces`:
```bash theme={null}
talosctl gen config my-cluster https://192.168.121.100:6443 --install-disk /dev/vda
```
Edit `controlplane.yaml` to add the virtual IP you picked to a network interface under `.machine.network.interfaces`, for example:
```yaml theme={null}
machine:
network:
interfaces:
- deviceSelector:
physical: true # should select any hardware network device, if you have just one, it will be selected
dhcp: true
vip:
ip: 192.168.121.100
```
Generate a machine configuration:
```bash theme={null}
talosctl gen config my-cluster https://192.168.121.100:6443 --install-disk /dev/vda --config-patch-control-plane @patch.yaml
```
Apply the configuration to the initial control plane node:
```bash theme={null}
talosctl -n 192.168.121.203 apply-config --insecure --file controlplane.yaml
```
You can tail the logs of the node:
```bash theme={null}
sudo tail -f /tmp/control-plane-node-1.log
```
Set up your shell to use the generated talosconfig and configure its endpoints (use the IPs of the control plane nodes):
```bash theme={null}
export TALOSCONFIG=$(realpath ./talosconfig)
talosctl config endpoint 192.168.121.203 192.168.121.119 192.168.121.125
```
Bootstrap the Kubernetes cluster from the initial control plane node:
```bash theme={null}
talosctl -n 192.168.121.203 bootstrap
```
Finally, apply the machine configurations to the remaining nodes:
```bash theme={null}
talosctl -n 192.168.121.119 apply-config --insecure --file controlplane.yaml
talosctl -n 192.168.121.125 apply-config --insecure --file controlplane.yaml
talosctl -n 192.168.121.69 apply-config --insecure --file worker.yaml
```
After a while, you should see that all the members have joined:
```bash theme={null}
talosctl -n 192.168.121.203 get members
```
The output will be like the following:
```text theme={null}
NODE NAMESPACE TYPE ID VERSION HOSTNAME MACHINE TYPE OS ADDRESSES
192.168.121.203 cluster Member talos-192-168-121-119 1 talos-192-168-121-119 controlplane Talos (v1.1.0) ["192.168.121.119"]
192.168.121.203 cluster Member talos-192-168-121-69 1 talos-192-168-121-69 worker Talos (v1.1.0) ["192.168.121.69"]
192.168.121.203 cluster Member talos-192-168-121-203 6 talos-192-168-121-203 controlplane Talos (v1.1.0) ["192.168.121.100","192.168.121.203"]
192.168.121.203 cluster Member talos-192-168-121-125 1 talos-192-168-121-125 controlplane Talos (v1.1.0) ["192.168.121.125"]
```
## Interacting with Kubernetes cluster
Retrieve the kubeconfig from the cluster:
```bash theme={null}
talosctl -n 192.168.121.203 kubeconfig ./kubeconfig
```
List the nodes in the cluster:
```bash theme={null}
kubectl --kubeconfig ./kubeconfig get node -owide
```
You will see an output similar to:
```text theme={null}
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
talos-192-168-121-203 Ready control-plane,master 3m10s v1.24.2 192.168.121.203 Talos (v1.1.0) 5.15.48-talos containerd://1.6.6
talos-192-168-121-69 Ready 2m25s v1.24.2 192.168.121.69 Talos (v1.1.0) 5.15.48-talos containerd://1.6.6
talos-192-168-121-119 Ready control-plane,master 8m46s v1.24.2 192.168.121.119 Talos (v1.1.0) 5.15.48-talos containerd://1.6.6
talos-192-168-121-125 Ready control-plane,master 3m11s v1.24.2 192.168.121.125 Talos (v1.1.0) 5.15.48-talos containerd://1.6.6
```
Congratulations, you have a highly-available Talos cluster running!
## Cleanup
You can destroy the vagrant environment by running:
```bash theme={null}
vagrant destroy -f
```
And remove the ISO image you downloaded:
```bash theme={null}
sudo rm -f /tmp/metal-amd64.iso
```
# VMware
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/virtualized-platforms/vmware
Creating Talos Kubernetes cluster using VMware.
## Creating a cluster via the `govc` CLI
In this guide we will create an HA Kubernetes cluster with 2 worker nodes.
We will use the `govc` cli which can be downloaded [here](https://github.com/vmware/govmomi/tree/master/govc#installation).
## Prerequisites
This guide will use the virtual IP ("VIP") functionality that is built into Talos in order to provide a stable, known IP for the Kubernetes control plane.
This simply means the user should pick an IP on their "VM Network" to designate for this purpose and keep it handy for future steps.
The Talos OVA image uses the `vmx-15` hardware version, this requires a minimum of **ESXi 6.7U2**.
To check your version of ESXi refer to the following Broadcom
[KB article](https://knowledge.broadcom.com/external/article/316595/build-numbers-and-versions-of-vmware-esx.html).
More information regarding virtual machine hardware versions can be found in the following Broadcom
[KB article](https://knowledge.broadcom.com/external/article/315655/virtual-machine-hardware-versions.html).
## Create the machine configuration files
### Generating base configurations
Using the VIP chosen in the prereq steps, we will now generate the base configuration files for the Talos machines.
This can be done with the `talosctl gen config ...` command.
Take note that we will also use a JSON6902 patch when creating the configs so that the control plane nodes get some special information about the VIP we chose earlier, as well as a daemonset to install vmware tools on talos nodes.
First, download `cp.patch.yaml` to your local machine and edit the VIP to match your chosen IP.
You can do this by issuing:
It's contents should look like the following:
```yaml theme={null}
- op: add
path: /machine/network
value:
interfaces:
- interface: eth0
dhcp: true
vip:
ip:
```
With the patch in hand, generate machine configs with:
```bash theme={null}
$ talosctl gen config vmware-test https://: --config-patch-control-plane @cp.patch.yaml
created controlplane.yaml
created worker.yaml
created talosconfig
```
At this point, you can modify the generated configs to your liking if needed.
Optionally, you can specify additional patches by adding to the `cp.patch.yaml` file downloaded earlier, or create your own patch files.
### Validate the configuration files
```bash theme={null}
$ talosctl validate --config controlplane.yaml --mode cloud
controlplane.yaml is valid for cloud mode
$ talosctl validate --config worker.yaml --mode cloud
worker.yaml is valid for cloud mode
```
> Note: Using VMXNET network interfaces in VMware will cause the default [Flannel CNI](https://github.com/flannel-io/flannel) backend (vxlan) to not work between nodes.
> To avoid this problem it is recommended to use Intel e1000 network interfaces or apply the patch below to [disable incompatible features](https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md).
Add the following configuration to disable incompatible hardware acceleration features.
```yaml theme={null}
apiVersion: v1alpha1
kind: EthernetConfig
name: flannel.1
features:
tx-checksum-ip-generic: false
```
## Set environment variables
`govc` makes use of the following environment variables
```bash theme={null}
export GOVC_URL=
export GOVC_USERNAME=
export GOVC_PASSWORD=
```
> Note: If your vCenter installation makes use of self signed certificates, you'll want to export `GOVC_INSECURE=true`.
There are some additional variables that you may need to set:
```bash theme={null}
export GOVC_DATACENTER=
export GOVC_RESOURCE_POOL=
export GOVC_DATASTORE=
export GOVC_NETWORK=
```
## Choose install approach
As part of this guide, we have a more automated install script that handles some of the complexity of importing OVAs and creating VMs.
If you wish to use this script, we will detail that next.
If you wish to carry out the manual approach, simply skip ahead to the "Manual Approach" section.
### Scripted install
Download the `vmware.sh` script to your local machine.
You can do this by issuing:
This script has default variables for things like Talos version and cluster name that may be interesting to tweak before deploying.
The script downloads VMWare OVA with `talos-vmtoolsd` from
Image Factory
extension pre-installed.
#### Import OVA
To create a content library and import the Talos OVA corresponding to the mentioned Talos version, simply issue:
```bash theme={null}
./vmware.sh upload_ova
```
#### Create cluster
With the OVA uploaded to the content library, you can create a 5 node (by default) cluster with 3 control plane and 2 worker nodes:
```bash theme={null}
./vmware.sh create
```
This step will create a VM from the OVA, edit the settings based on the env variables used for VM size/specs, then power on the VMs.
You may now skip past the "Manual Approach" section down to "Bootstrap Cluster".
### Manual approach
#### Import the OVA into vCenter
A `talos.ova` asset is available from [Image Factory](https://www.talos.dev/latest/talos-guides/install/boot-assets/#image-factory).
We will refer to the version of the release as `$TALOS_VERSION` below.
It can be easily exported with `export TALOS_VERSION="v0.3.0-alpha.10"` or similar.
The download link already includes the `talos-vmtoolsd` extension.
```bash theme={null}
curl -LO https://factory.talos.dev/image/903b2da78f99adef03cbbd4df6714563823f63218508800751560d3bc3557e40/${TALOS_VERSION}/vmware-amd64.ova
```
Create a content library (if needed) with:
```bash theme={null}
govc library.create
```
Import the OVA to the library with:
```bash theme={null}
govc library.import -n talos-${TALOS_VERSION} /path/to/downloaded/talos.ova
```
#### Create the bootstrap node
We'll clone the OVA to create the bootstrap node (our first control plane node).
```bash theme={null}
govc library.deploy /talos-${TALOS_VERSION} control-plane-1
```
Talos makes use of the `guestinfo` facility of VMware to provide the machine/cluster configuration.
This can be set using the `govc vm.change` command.
To facilitate persistent storage using the vSphere cloud provider integration with Kubernetes, `disk.enableUUID=1` is used.
```bash theme={null}
govc vm.change \
-e "guestinfo.talos.config=$(cat controlplane.yaml | base64)" \
-e "disk.enableUUID=1" \
-vm control-plane-1
```
#### Update hardware resources for the bootstrap node
* `-c` is used to configure the number of cpus
* `-m` is used to configure the amount of memory (in MB)
```bash theme={null}
govc vm.change \
-c 2 \
-m 4096 \
-vm control-plane-1
```
The following can be used to adjust the [EPHEMERAL](../../learn-more/architecture#file-system-partitions) disk size.
```bash theme={null}
govc vm.disk.change -vm control-plane-1 -disk.name disk-1000-0 -size 10G
```
```bash theme={null}
govc vm.power -on control-plane-1
```
#### Create the remaining control plane nodes
```bash theme={null}
govc library.deploy /talos-${TALOS_VERSION} control-plane-2
govc vm.change \
-e "guestinfo.talos.config=$(base64 controlplane.yaml)" \
-e "disk.enableUUID=1" \
-vm control-plane-2
govc library.deploy /talos-${TALOS_VERSION} control-plane-3
govc vm.change \
-e "guestinfo.talos.config=$(base64 controlplane.yaml)" \
-e "disk.enableUUID=1" \
-vm control-plane-3
```
```bash theme={null}
govc vm.change \
-c 2 \
-m 4096 \
-vm control-plane-2
govc vm.change \
-c 2 \
-m 4096 \
-vm control-plane-3
```
```bash theme={null}
govc vm.disk.change -vm control-plane-2 -disk.name disk-1000-0 -size 10G
govc vm.disk.change -vm control-plane-3 -disk.name disk-1000-0 -size 10G
```
```bash theme={null}
govc vm.power -on control-plane-2
govc vm.power -on control-plane-3
```
#### Update settings for the worker nodes
```bash theme={null}
govc library.deploy /talos-${TALOS_VERSION} worker-1
govc vm.change \
-e "guestinfo.talos.config=$(base64 worker.yaml)" \
-e "disk.enableUUID=1" \
-vm worker-1
govc library.deploy /talos-${TALOS_VERSION} worker-2
govc vm.change \
-e "guestinfo.talos.config=$(base64 worker.yaml)" \
-e "disk.enableUUID=1" \
-vm worker-2
```
```bash theme={null}
govc vm.change \
-c 4 \
-m 8192 \
-vm worker-1
govc vm.change \
-c 4 \
-m 8192 \
-vm worker-2
```
```bash theme={null}
govc vm.disk.change -vm worker-1 -disk.name disk-1000-0 -size 10G
govc vm.disk.change -vm worker-2 -disk.name disk-1000-0 -size 10G
```
```bash theme={null}
govc vm.power -on worker-1
govc vm.power -on worker-2
```
#### Bootstrap cluster
In the vSphere UI, open a console to one of the control plane nodes.
You should see some output stating that etcd should be bootstrapped.
This text should look like:
```bash theme={null}
"etcd is waiting to join the cluster, if this node is the first node in the cluster, please run `talosctl bootstrap` against one of the following IPs:
```
Take note of the IP mentioned here and issue:
```bash theme={null}
talosctl --talosconfig talosconfig bootstrap -e -n
```
Keep this IP handy for the following steps as well.
#### Retrieve the `kubeconfig`
At this point we can retrieve the admin `kubeconfig` by running:
```bash theme={null}
talosctl --talosconfig talosconfig config endpoint
talosctl --talosconfig talosconfig config node
talosctl --talosconfig talosconfig kubeconfig .
```
#### Configure `talos-vmtoolsd`
The talos-vmtoolsd application was deployed as a daemonset as part of the cluster creation; however, we must now provide a talos credentials file for it to use.
Create a new talosconfig with:
```bash theme={null}
talosctl --talosconfig talosconfig -n config new vmtoolsd-secret.yaml --roles os:admin
```
Create a secret from the talosconfig:
```bash theme={null}
kubectl -n kube-system create secret generic talos-vmtoolsd-config \
--from-file=talosconfig=./vmtoolsd-secret.yaml
```
Clean up the generated file from local system:
```bash theme={null}
rm vmtoolsd-secret.yaml
```
Once configured, you should now see these daemonset pods go into "Running" state and in vCenter, you will now see IPs and info from the Talos nodes present in the UI.
# Xen
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/virtualized-platforms/xen
Talos is known to work on Xen.
We don't yet have a documented guide specific to Xen; however, you can follow the [General Getting Started Guide](../../getting-started/getting-started).
If you run into any issues, our [slack community](https://taloscommunity.slack.com/) can probably help!
If you are not already a member, you can request access [here](https://inviter.co/sidero-labs-community).
> Note: For Secure Boot, you can force setup mode with `varstore-sb-state setup` or `xe vm-set-uefi-mode mode=setup uuid=`.
> Don't forget to re-enable Secure Boot after the first boot.
# Xen Orchestra
Source: https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/virtualized-platforms/xenorchestra
Creating Talos Kubernetes cluster using Xen Orchestra.
In this guide, we will create a Kubernetes cluster using Xen Orchestra.
This guide does not cover XCP-ng without Xen Orchestra installed, or XO Lite, even though some of the steps may be similar.
## Installation
### How to get Xen Orchestra
It is assumed that you already have XCP-ng and Xen Orchestra installed.
If necessary, visit the [XCP-ng downloads page](https://xcp-ng.org/) and [Xen Orchestra documentation](https://xen-orchestra.com/).
### Install talosctl
You can download `talosctl` on MacOS and Linux via:
```bash theme={null}
brew install siderolabs/tap/talosctl
```
For manual installation and other platforms, see the [talosctl installation guide](../../getting-started/talosctl).
### Download the RAW disk image
To install Talos in Xen Orchestra, you will need the RAW image from [Image Factory](../boot-assets/#image-factory). To work with XCP-ng the image needs to be built for the Nocloud platform with the system extension `siderolabs/xen-guest-agent`. See this example Talos image you can use.
For example version for `linux`, download and extract the image:
**Xen Guest Agent**
It is recommended to use the Talos image with the system extension `siderolabs/xen-guest-agent`. It should work without it but some features, like IP addresses reported in Xen Orchestra, will be missing.
## Create the template VM
This section explains how to create a reusable Talos VM template in Xen Orchestra.
### Recommended configuration for VMs
Before you begin, familiarize yourself with Talos' [system requirements](../../getting-started/system-requirements) and allocate resources\
accordingly.
| Parameter | Recommended Value | Notes |
| ------------------------------- | ------------------------------------ | --------------------------------------------- |
| **Boot Mode** | UEFI | Modern firmware, Secure Boot support |
| **vCPUs** | 2+ (control plane), 4+ (workers) | Minimum 2 cores required |
| **Memory** | 4GB+ (control plane), 8GB+ (workers) | Minimum 2GB required |
| **Storage Repository** | Local Storage or Shared Storage | Depending on your infrastructure |
| **Cloud-Init** | Enabled | To automatically pass the Talos configuration |
| **Advanced settings > Boot VM** | false | To not start the template VM after creation |
### Via Xen Orchestra web UI
Follow these steps to create a Talos template using the Xen Orchestra web interface.
#### Step 1: Upload disk image to Xen Orchestra
1. Log in to Xen Orchestra.
2. Navigate to (**Sidebar Menu** >) **Import** > **Disks**.
3. Select the destination Storage Repository (SR).
4. Drop the RAW file.
5. Click the blue button below which displays the name of the disk.
6. Change the **Name** to "talos-nocloud-amd64". Add a description.
7. Click **Import**.
8. Wait until the download is complete.
![Screenshot of XO5 UI showing the import of a disk image]()
#### Step 2: Create a new VM
Create a new VM with the following settings:
1. Click (**Sidebar Menu** >) **New** > **VM**.
2. Select the pool you want to create the VM on.
3. Select **Template** as **Generic Linux UEFI**.
4. Set **Name** to (for the control plane).
5. Configure the following:
**System configuration**
* Boot firmware: **UEFI**
* vCPUs: 2 minimum (4+ recommended for workers)
* RAM: 4096 MB minimum (4GB)
* Topology: (Default)
**Install settings**
* Select PXE or any ISO/DVD, we won’t use it
**Network**
* Select your network
**Disk**
* Do not create a new disk
* We will attach the imported Talos VDI
**Advanced options**
* Boot firmware : **UEFI**
* Boot VM after creation : Off
6. Click on **Create**
![Screenshot of XO5 UI showing the creation of a new VM to use as a template]()
#### Step 3: Attach the Talos VDI
After creating the VM, attach the Talos disk image:
1. If the VM page isn't automatically open after creation, navigate to (**Sidebar Menu** >) **Home** > **VMs**, clear the filters, and type .
2. Open the **Disk** tab.
3. Remove any existing disk.
4. Click **Attach disk**.
5. Select **talos-nocloud-amd64**.
6. Click **Attach** below the dropdown field.
![Screenshot of XO5 UI showing the attachment of the Talos disk to the template VM]()
#### Step 4: Finalize template creation
Complete the template setup:
1. Open the **Advanced** tab.
2. Change the **Boot order** to set the **Hard-Drive** first (instead of Network/PXE or DVD drive).
3. Verify that the VM boots by starting it and opening the **Console** tab.
![Screenshot of XO5 UI showing the template VM console]()
#### Step 5: Convert to template
Convert the VM to a template:
1. Stop the VM if it's running.
2. Open the **Advanced** tab.
3. Click **Convert to template** and confirm.
Navigate to (**Sidebar Menu** >) **Home** > **Templates** and search for "talos" to verify your new template appears.
![Screenshot of XO5 UI showing the list of templates containing our new Talos template]()
### Alternative
This template building can also be done using the [XCP-ng Packer.io builder](https://github.com/vatesfr/packer-plugin-xenserver).
## Create the Talos cluster
Now that you have a template, you can quickly create Talos VMs with different configurations using cloud-init.
In this guide we will create a simple Kubernetes cluster with one control plane and one worker node.
### Prerequisites
This guide uses the virtual IP (VIP) functionality built into Talos to provide a stable IP for the Kubernetes control plane. Choose an IP on your VM network to designate for this purpose and note it for use in the following steps.
### Step 1: Generate Talos configurations
Generate the base configuration files for the Talos machines using the control plane IP. We will use a patch to provide the control plane nodes with the virtual IP information.
Create `cp.patch.yaml` with the content:
```yaml cp.patch.yaml icon="file-code" theme={null}
- op: add
path: /machine/network
value:
interfaces:
- interface: enX0
dhcp: true
vip:
ip:
```
In order to use static IP addresses for both the control plane and the worker, this patch needs to be edited and an additional patch needs to be created for the worker configuration.
Read the [related Talos documentation](https://docs.siderolabs.com/talos/v1.8/configure-your-talos-cluster/system-configuration/patching#machine-network).
With the patch in hand, generate machine configs with:
```shell theme={null}
export VIP_IP=
export PORT=6443
```
Generate the configurations:
```shell theme={null}
talosctl gen config talos-xcp-cluster https://${VIP_IP}:${PORT} --config-patch-control-plane @cp.yaml
```
This creates:
* `controlplane.yaml` - Configuration for control planes
* `worker.yaml` - Configuration for workers
* `talosconfig` - Talosctl configuration file
You can modify the generated configurations if needed. You can also specify additional patches by adding to the `cp.patch.yaml` file created earlier or by creating your own patch files.
Validate the configuration files:
```shell theme={null}
talosctl validate --config controlplane.yaml --mode cloud
talosctl validate --config worker.yaml --mode cloud
```
### Step 2: Create the control plane from the template
Create the control plane VM using the Xen Orchestra web interface.
#### Via Xen Orchestra web UI
Create the control plane VM:
1. Click **+ New VM**.
2. Select the pool you want to create the VM on.
3. In the **Info** section:
* Select the template .
* Change the **Name** to something like `demo-talos-cp`.
4. In the **Performance** section:
* Adjust VM settings according to requirements.
5. In the **Install settings** section:
* Click **Custom config**.
* Paste the content of `controlplane.yaml` into **User config**.
* Add at least a comment `#` in **Network config** to avoid form validation errors.
6. In the **Interfaces** section:
* Select your network.
7. In the **Disks** section:
* Expand disk size to match your needs.
8. Click **Create**.
![Screenshot of XO5 UI showing the creation of a new VM for the control plane]()
The VM will start and automatically apply the Talos configuration via cloud-init!
Once the machine has started, check the VM dashboard, int the **Network** tab: the guest agent should have reported the VM IP address.
Take note of this IP address, which will be referred to as `$CONTROL_PLANE_IP` for the rest of this guide.
If you wish to export this IP as a bash variable, simply issue a command like `export CONTROL_PLANE_IP=1.2.3.4`
#### Alternative with Terraform
The [Terraform provider for Xen Orchestra](https://registry.terraform.io/providers/vatesfr/xenorchestra/latest) can be used as an alternative to the Web UI, enabling declarative and automated infrastructure management.
An example is provided in the contrib repository: [Xen Orchestra Terraform examples](https://github.com/siderolabs/contrib/tree/main/examples/terraform/xenorchestra).
### Step 3: Create the worker node
Create at least one worker node using a process similar to the control plane creation above. Use the `worker.yaml` configuration file in the **Install settings** > **User config** section.
Take note of the worker node’s IP address, which will be referred to as `$WORKER_IP`
### Step 4: Bootstrap cluster
Open a console to the control plane node in the Xen Orchestra UI. You should see output indicating that etcd needs to be bootstrapped:
```text theme={null}
"etcd is waiting to join the cluster, if this node is the first node in the cluster, please run talosctl bootstrap against one of the following IPs:
```
#### Configure `talosctl`
Set up the talosctl configuration:
```bash theme={null}
export TALOSCONFIG="talosconfig"
export CONTROL_PLANE_IP=
talosctl config endpoint ${CONTROL_PLANE_IP}
talosctl config node ${CONTROL_PLANE_IP}
```
Check the status of the nodes:
```bash theme={null}
talosctl version --nodes ${CONTROL_PLANE_IP}
talosctl services --nodes ${CONTROL_PLANE_IP} | grep etcd
talosctl services --nodes ${CONTROL_PLANE_IP} | grep xen-guest-agent
```
If it does not appear, the node is not yet in a `bootstrappable` state.
#### Bootstrap etcd
Initialize the Etcd cluster:
```bash theme={null}
talosctl bootstrap --nodes ${CONTROL_PLANE_IP}
```
Wait a few minutes for the cluster to stabilize.
![Screenshot of XO6 UI showing the console of the control plane VM running]()
#### Retrieve the `kubeconfig`
Once the bootstrap is complete, retrieve the admin `kubeconfig`:
```bash theme={null}
talosctl kubeconfig --nodes ${CONTROL_PLANE_IP}
```
This creates a `kubeconfig` file in the current directory.
Verify the Kubernetes cluster:
```bash theme={null}
export KUBECONFIG=./kubeconfig
kubectl get nodes -w
```
## Secure Boot
For Secure Boot setup, see the **Secure Boot guide** in the [Talos documentation](../../platform-specific-installations/bare-metal-platforms/secureboot) and **Enable Secure Boot for a Guest VM** in the [XCP-ng documentation](https://docs.xcp-ng.org/guides/guest-UEFI-Secure-Boot/#enable-secure-boot-for-a-guest-vm).
The easiest way to get started with Secure Boot is to download the Secure Boot ISO and boot it on a UEFI-enabled VM with Secure Boot enabled in setup mode.
The installation should be performed using the Secure Boot installer image. Use this value in your Talos machine configuration:
You can use the **Generic Linux UEFI** template to create the VM in Xen Orchestra. Force setup mode with one of these commands:
```bash theme={null}
varstore-sb-state setup
```
or
```bash theme={null}
xe vm-set-uefi-mode mode=setup uuid=
```
The ISO bootloader will enroll the keys in the UEFI firmware and boot Talos Linux in Secure Boot mode.
Once Talos is running in maintenance mode, verify that Secure Boot is enabled:
```bash theme={null}
$ talosctl -n $CONTROL_PLANE_IP get securitystate --insecure
NODE NAMESPACE TYPE ID VERSION SECUREBOOT
runtime SecurityState securitystate 1 true
```
Generate the machine configuration for the node using the Secure Boot installer image:
Apply the machine configuration to the node:
```bash theme={null}
talosctl -n ${CONTROL_PLANE_IP} apply-config --insecure -f controlplane.yaml
```
Talos will install to the disk and reboot the node. Ensure the ISO image is no longer attached to the node to prevent booting from the ISO again.
Once the node is rebooted, verify Secure Boot mode:
```bash theme={null}
talosctl -n ${CONTROL_PLANE_IP} --talosconfig=talosconfig get securitystate
```
## Cloud Controller Manager (CCM) for Xen Orchestra
To install the Cloud Controller Manager in this cluster, refer to the [Xen Orchestra CCM documentation](https://github.com/vatesfr/xenorchestra-cloud-controller-manager/).
## References
* [Talos Documentation](https://www.talos.dev/)
* [XCP-ng](https://docs.xcp-ng.org/) [Documentation](https://docs.xcp-ng.org/)
* [Xen Orchestra Documentation](https://xen-orchestra.com/docs/)
* [Talos Image Factory](https://factory.talos.dev/)
* [Talos System Requirements](https://www.talos.dev/latest/talos-guides/install/system-requirements/)
# CA Rotation
Source: https://docs.siderolabs.com/talos/v1.13/security/ca-rotation
How to rotate Talos and Kubernetes API root certificate authorities.
In general, you almost never need to rotate the root CA certificate and key for the Talos API and Kubernetes API.
Talos sets up root certificate authorities with the lifetime of 10 years, and all Talos and Kubernetes API certificates are issued by these root CAs.
So the rotation of the root CA is only needed if:
* you suspect that the private key has been compromised;
* you want to revoke access to the cluster for a leaked `talosconfig` or `kubeconfig`;
* once in 10 years.
## Overview
There are some details which make Talos and Kubernetes API root CA rotation a bit different, but the general flow is the same:
* generate new CA certificate and key;
* add new CA certificate as 'accepted', so new certificates will be accepted as valid;
* swap issuing CA to the new one, old CA as accepted;
* refresh all certificates in the cluster;
* remove old CA from 'accepted'.
At the end of the flow, old CA is completely removed from the cluster, so all certificates issued by it will be considered invalid.
Both rotation flows are described in detail below.
## Talos API
### Automated Talos API CA rotation
Talos API CA rotation doesn't interrupt connections within the cluster, and it doesn't require a reboot of the nodes.
Run the following command in dry-run mode to see the steps which will be taken:
```shell theme={null}
$ talosctl -n rotate-ca --dry-run=true --talos=true --kubernetes=false
> Starting Talos API PKI rotation, dry-run mode true...
> Using config context: "talos-default"
> Using Talos API endpoints: ["172.20.0.2"]
> Cluster topology:
- control plane nodes: ["172.20.0.2"]
- worker nodes: ["172.20.0.3"]
> Current Talos CA:
...
```
No changes will be done to the cluster in dry-run mode, so you can safely run it to see the steps.
Before proceeding, make sure that you can capture the output of `talosctl` command, as it will contain the new CA certificate and key.
Record a list of Talos API users to make sure they can all be updated with new `talosconfig`.
Run the following command to rotate the Talos API CA:
Once the rotation is done, stash the new Talos CA, update `secrets.yaml` (if using that for machine configuration generation) with new CA key and certificate.
The new client `talosconfig` is written to the current directory as `talosconfig`.
You can merge it to the default location with `talosctl config merge ./talosconfig`.
If other client access `talosconfig` files needs to be generated, use `talosctl config new` with new `talosconfig`.
> Note: if using [Talos API access from Kubernetes](../../../kubernetes-guides/advanced-guides/talos-api-access-from-k8s) feature, pods might need to be restarted manually to pick up new `talosconfig`.
### Manual steps for Talos API CA rotation
1. Generate new Talos CA (e.g. use `talosctl gen secrets` and use Talos CA).
2. Patch machine configuration on all nodes updating `.machine.acceptedCAs` with new CA certificate.
3. Generate `talosconfig` with client certificate generated with new CA, but still using old CA as server CA, verify connectivity, Talos should accept new client certificate.
4. Patch machine configuration on all nodes updating `.machine.ca` with new CA certificate and key, and keeping old CA certificate in `.machine.acceptedCAs` (on worker nodes `.machine.ca` doesn't have the key).
5. Generate `talosconfig` with both client certificate and server CA using new CA PKI, verify connectivity.
6. Remove old CA certificate from `.machine.acceptedCAs` on all nodes.
7. Verify connectivity.
## Kubernetes API
### Automated Kubernetes API CA rotation
The automated process only rotates Kubernetes API CA, used by the `kube-apiserver`, `kubelet`, etc.
Other Kubernetes secrets might need to be rotated manually as required.
Kubernetes pods might need to be restarted to handle changes, and communication within the cluster might be disrupted during the rotation process.
Run the following command in dry-run mode to see the steps which will be taken:
```shell theme={null}
$ talosctl -n rotate-ca --dry-run=true --talos=false --kubernetes=true
> Starting Kubernetes API PKI rotation, dry-run mode true...
> Cluster topology:
- control plane nodes: ["172.20.0.2"]
- worker nodes: ["172.20.0.3"]
> Building current Kubernetes client...
> Current Kubernetes CA:
...
```
Before proceeding, make sure that you can capture the output of `talosctl` command, as it will contain the new CA certificate and key.
As Talos API access will not be disrupted, the changes can be reverted back if needed by reverting machine configuration.
Run the following command to rotate the Kubernetes API CA:
```shell theme={null}
$ talosctl -n rotate-ca --dry-run=false --talos=false --kubernetes=true
> Starting Kubernetes API PKI rotation, dry-run mode false...
> Cluster topology:
- control plane nodes: ["172.20.0.2"]
- worker nodes: ["172.20.0.3"]
> Building current Kubernetes client...
> Current Kubernetes CA:
...
> New Kubernetes CA:
...
> Verifying connectivity with existing PKI...
- OK (2 nodes ready)
> Adding new Kubernetes CA as accepted...
- 172.20.0.2: OK
- 172.20.0.3: OK
> Making new Kubernetes CA the issuing CA, old Kubernetes CA the accepted CA...
- 172.20.0.2: OK
- 172.20.0.3: OK
> Building new Kubernetes client...
> Verifying connectivity with new PKI...
2024/04/17 21:45:52 retrying error: Get "https://172.20.0.1:6443/api/v1/nodes": EOF
- OK (2 nodes ready)
> Removing old Kubernetes CA from the accepted CAs...
- 172.20.0.2: OK
- 172.20.0.3: OK
> Verifying connectivity with new PKI...
- OK (2 nodes ready)
> Kubernetes CA rotation done, new 'kubeconfig' can be fetched with `talosctl kubeconfig`.
```
At the end of the process, Kubernetes control plane components will be restarted to pick up CA certificate changes.
Each node `kubelet` will re-join the cluster with new client certificate.
New `kubeconfig` can be fetched with `talosctl kubeconfig` command from the cluster.
Kubernetes pods might need to be restarted manually to pick up changes to the Kubernetes API CA.
### Manual steps for Kubernetes API CA rotation
Steps are similar [to the Talos API CA rotation](#manual-steps-for-talos-api-ca-rotation), but use:
* `.cluster.acceptedCAs` in place of `.machine.acceptedCAs`;
* `.cluster.ca` in place of `.machine.ca`;
* `kubeconfig` in place of `talosconfig`.
# How to manage PKI and certificate lifetimes with Talos Linux
Source: https://docs.siderolabs.com/talos/v1.13/security/cert-management
Manage certificate lifetimes and regenerate client credentials in a Talos cluster.
Talos Linux automatically manages and rotates all server side certificates for etcd, Kubernetes, and the Talos API.
Note however that the kubelet needs to be restarted at least once a year in order for the certificates to be rotated.
Any upgrade/reboot of the node will suffice for this effect.
You can check the Kubernetes certificates with the command `talosctl get KubernetesDynamicCerts -o yaml` on the controlplane.
Client certificates (`talosconfig` and `kubeconfig`) are the user's responsibility.
Each time you download the `kubeconfig` file from a Talos Linux cluster, the client certificate is regenerated giving you a kubeconfig which is valid for a year.
The `talosconfig` file should be renewed at least once a year, using the `talosctl config new` command, as shown below, or by one of the other methods.
## Generating new client configuration
### Using control plane node
If you have a valid (not expired) `talosconfig` with `os:admin` role,
a new client configuration file can be generated with `talosctl config new` against
any controlplane node:
```shell theme={null}
talosctl -n CP1 config new talosconfig-reader --roles os:reader --crt-ttl 24h
```
A specific [role](./rbac) and certificate lifetime can be specified.
### From secrets bundle
If a secrets bundle (`secrets.yaml` from `talosctl gen secrets`) was saved while
[generating machine configuration](../getting-started/#configure-talos):
```shell theme={null}
talosctl gen config --with-secrets secrets.yaml --output-types talosconfig -o talosconfig https://
```
> Note: `` and `` arguments don't matter, as they are not used for `talosconfig`.
### From control plane machine configuration
In order to create a new key pair for client configuration, you will need the root Talos API CA.
The base64 encoded CA can be found in the control plane node's configuration file.
Save the CA public key, and CA private key as `ca.crt`, and `ca.key` respectively:
```shell theme={null}
yq eval .machine.ca.crt controlplane.yaml | base64 -d > ca.crt
yq eval .machine.ca.key controlplane.yaml | base64 -d > ca.key
```
Now, run the following commands to generate a certificate:
```bash theme={null}
talosctl gen key --name admin
talosctl gen csr --key admin.key --ip 127.0.0.1
talosctl gen crt --ca ca --csr admin.csr --name admin
```
Put the base64-encoded files to the respective location to the `talosconfig`:
```yaml theme={null}
context: mycluster
contexts:
mycluster:
endpoints:
- CP1
- CP2
ca:
crt:
key:
```
# Custom Certificate Authorities
Source: https://docs.siderolabs.com/talos/v1.13/security/certificate-authorities
How to supply custom certificate authorities
## Appending the Certificate Authority (CA)
Append additional certificate authorities to the system's trusted certificate store by [patching](../configure-your-talos-cluster/system-configuration/patching) the machine configuration with the following
[document](../reference/configuration/security/trustedrootsconfig):
```yaml theme={null}
apiVersion: v1alpha1
kind: TrustedRootsConfig
name: custom-ca
certificates: |-
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
```
Multiple documents can be appended, and multiple CA certificates might be present in each configuration document.
This configuration can be also applied in maintenance mode.
Please note that if the `STATE` partition is encrypted, the CA certificates will only be loaded after the partition is unlocked.
So the encryption method should allow unlocking the partition without the need for a CA certificate.
# IRSA with Talos Linux
Source: https://docs.siderolabs.com/talos/v1.13/security/iam-roles-for-service-accounts
How to enable IAM Roles for Service Accounts (IRSA) on Talos Linux.
This guide explains how to implement IAM Roles for Service Accounts (IRSA) for a Kubernetes cluster on Talos Linux. This allows Pods to access AWS services using temporary IAM credentials, similar to how EKS clusters in AWS function.
While this example uses AWS S3 and Terraform, the general principles can be applied to any S3-compatible object store and adapted for other infrastructure-as-code tools or a manual setup. For the purpose of this guide, we will create a generic, read-only IRSA Role to test access to S3 from Talos Linux after the IRSA setup is complete. Administrators will need to create their own IAM Roles that target the specific AWS services they need to access from their Kubernetes Pods.
## Prerequisites
Before you begin, you will need:
* A Kubernetes cluster running on Talos Linux.
* An AWS account with permissions to create S3 buckets, IAM roles, and OIDC providers.
* [Terraform](https://www.terraform.io/) (optional, for the examples provided).
* `kubectl` and `helm` installed locally for testing
This guide is based on the official instructions for setting up the [Amazon EKS Pod Identity Webhook in a self-hosted environment](https://github.com/aws/amazon-eks-pod-identity-webhook/blob/master/SELF_HOSTED_SETUP.md).
In summary, this is what is needed to accomplish IRSA:
1. Create a keypair, populate an S3 bucket with publicly-accessible OIDC discovery documents, and
register the S3 bucket endpoint as an Identity Provider with AWS IAM.
2. Pass the private key to the Kubernetes API server. Your Kubernetes API server is now an OIDC issuer!
3. Run `amazon-eks-pod-identity-webhook`, which mutates pods based on the annotations to inject
environment variables like `AWS_WEB_IDENTITY_TOKEN_FILE` (and `cert-manager`, since this webhook
per Kubernetes requirements needs to serve HTTPS)
The Kubernetes API server can now sign projected service account tokens using the private key,
which gets passed to AWS IAM with the role the pod wishes to assume. IAM then confirms the signature
using the registered IdP, ensures the role assumption is allowed, and returns credentials.
## Step 1: Create OIDC serving infrastructure
The following Terraform module creates all the AWS infrastructure needed. You can instantiate this module in your terraform using `source`, or apply it using a `.tfvars` file or command-line flags.
```hcl theme={null}
variable "bucket_name" {
type = string
}
variable "cluster_name" {
type = string
}
variable "secret_prefix" {
type = string
}
locals {
issuer_hostpath = "s3.${data.aws_region.current.region}.amazonaws.com/${var.bucket_name}"
openid_configuration = jsonencode({
issuer = "https://${local.issuer_hostpath}"
jwks_uri = "https://${local.issuer_hostpath}/keys.json"
authorization_endpoint = "urn:kubernetes:programmatic_authorization"
response_types_supported = ["id_token"]
subject_types_supported = ["public"]
id_token_signing_alg_values_supported = ["RS256"]
claims_supported = ["sub", "iss"]
})
# the JWKS format and encodings are defined in the RFC
# https://datatracker.ietf.org/doc/html/rfc7517
jwks = jsonencode({
keys = [
{
use = "sig"
alg = "RS256"
kty = "RSA"
kid = data.external.pub_der.result.der
n = data.external.modulus.result.modulus
e = "AQAB"
}]
})
}
data "aws_region" "current" {}
data "aws_caller_identity" "current" {}
# generate a key pair that will be used to sign projected service account tokens
resource "tls_private_key" "key" {
algorithm = "RSA"
rsa_bits = 2048
}
# this needs to go into talos machine configuration under cluster.serviceAccount.key
resource "aws_secretsmanager_secret" "signing_key" {
name = "${var.secret_prefix}/${var.cluster_name}"
}
resource "aws_secretsmanager_secret_version" "signing_key" {
secret_id = aws_secretsmanager_secret.signing_key.id
secret_string = base64encode(tls_private_key.key.private_key_pem)
}
# bucket that we're using as an OIDC discovery endpoint
resource "aws_s3_bucket" "oidc" {
bucket = var.bucket_name
}
# registering the public bucket host as an IdP
resource "aws_iam_openid_connect_provider" "oidc" {
url = "https://${local.issuer_hostpath}"
client_id_list = ["sts.amazonaws.com"]
thumbprint_list = [data.tls_certificate.oidc.certificates[0].sha1_fingerprint]
}
resource "aws_s3_bucket_ownership_controls" "oidc" {
bucket = aws_s3_bucket.oidc.id
rule {
object_ownership = "BucketOwnerPreferred"
}
}
# we _want_ this bucket to have publicly accessible files
resource "aws_s3_bucket_public_access_block" "oidc" {
bucket = aws_s3_bucket.oidc.id
block_public_acls = false
block_public_policy = false
ignore_public_acls = false
restrict_public_buckets = false
}
# the two files we need to serve
resource "aws_s3_object" "keys_json" {
bucket = aws_s3_bucket.oidc.id
key = "keys.json"
content = local.jwks
acl = "public-read"
etag = md5(local.jwks)
depends_on = [
aws_s3_bucket_ownership_controls.oidc,
aws_s3_bucket_public_access_block.oidc,
]
}
resource "aws_s3_object" "openid-configuration" {
bucket = aws_s3_bucket.oidc.id
key = ".well-known/openid-configuration"
content = local.openid_configuration
acl = "public-read"
etag = md5(local.openid_configuration)
depends_on = [
aws_s3_bucket_ownership_controls.oidc,
aws_s3_bucket_public_access_block.oidc,
]
}
data "tls_certificate" "oidc" {
url = "https://${local.issuer_hostpath}"
}
# This is used for the `kid` Key ID field in the JWKS, which is an arbitrary string that can uniquely
# identify a key.
# This logic comes from https://github.com/kubernetes/kubernetes/pull/78502. It creates unique and
# deterministic outputs across platforms.
# See also https://datatracker.ietf.org/doc/html/rfc4648#section-5 for final base64url encoding
data "external" "pub_der" {
program = ["bash", "-c", < machineconfig-patch.yaml
cluster:
apiServer:
extraArgs:
service-account-issuer: ${ISSUER_HOSTPATH}
serviceAccount:
key: ${BASE64_ENCODED_PRIVATE_KEY}
EOF
```
2. Apply the patch to your Talos configuration and update your cluster, and wait for the server to come back up with the new config. For example, using `talosctl`:
```bash theme={null}
talosctl apply-config --nodes --file machineconfig-patch.yaml
```
## Step 3: Install required Kubernetes components
Two components are required on the cluster: `cert-manager` and `amazon-eks-pod-identity-webhook`.
### Install `cert-manager`
1. Add the Jetstack Helm repository:
```bash theme={null}
helm repo add jetstack https://charts.jetstack.io
helm repo update
```
2. Install the `cert-manager` Helm chart:
```bash theme={null}
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--set crds.enabled=true \
--create-namespace
```
Next, install the `amazon-eks-pod-identity-webhook`.
### Install `amazon-eks-pod-identity-webhook`
1. Add the `jkroepke` Helm repository:
```bash theme={null}
helm repo add jkroepke https://jkroepke.github.io/helm-charts/
helm repo update
```
2. Install the `amazon-eks-pod-identity-webhook` Helm chart:
```bash theme={null}
helm install amazon-eks-pod-identity-webhook jkroepke/amazon-eks-pod-identity-webhook \
--namespace kube-system \
set config.defaultAwsRegion=${AWS_REGION}
```
## Step 4: Test
1. Apply this terraform (or do manually) to create an AWS role for a test service account to assume that has S3 read access.
```hcl theme={null}
resource "aws_iam_role" "talos_irsa_s3_readonly_example" {
name = var.service_account_name
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRoleWithWebIdentity"
Effect = "Allow"
Sid = ""
Principal = {
Federated = "arn:aws:iam::${var.aws_account_id}:oidc-provider/${var.issuer_hostname}"
}
Condition = {
StringEquals = {
"${var.issuer_hostname}:aud": "sts.amazonaws.com",
"${var.issuer_hostname}:sub": "system:serviceaccount:${var.namespace}:${var.service_account_name}"
}
}
}
]
})
}
resource "aws_iam_role_policy_attachment" "talos_irsa_s3_readonly_example" {
role = aws_iam_role.talos_irsa_s3_readonly_example.name
policy_arn = "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
}
```
2. Create a manifest for the `ServiceAccount` and a test `Pod`. The command below uses a `heredoc` to create `test-pod.yaml` and substitute the shell variables.
```bash theme={null}
cat < test-pod.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: ${SERVICEACCOUNT_NAME}
namespace: ${NAMESPACE}
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::${AWS_ACCOUNT}:role/${SERVICEACCOUNT_NAME}"
eks.amazonaws.com/sts-regional-endpoints: "true"
eks.amazonaws.com/token-expiration: "86400"
---
apiVersion: v1
kind: Pod
metadata:
namespace: ${NAMESPACE}
name: aws-cli
spec:
serviceAccountName: ${SERVICEACCOUNT_NAME}
containers:
- name: aws-cli
image: amazon/aws-cli:latest
command: ["sleep", "infinity"]
EOF
```
3. Apply the manifest:
```bash theme={null}
kubectl apply -f test-pod.yaml
```
4. Exec into the `aws-cli` `Pod`.and test access by listing S3 buckets:
```bash theme={null}
kubectl exec -it aws-cli -n ${NAMESPACE} -- aws s3 ls
```
This command should list the S3 buckets in your AWS account, confirming that IRSA is correctly configured.
# Machine Configuration OAuth2 Authentication
Source: https://docs.siderolabs.com/talos/v1.13/security/machine-config-oauth
How to authenticate Talos machine configuration download (`talos.config=`) on `metal` platform using OAuth.
Talos Linux when running on the `metal` platform can be configured to authenticate the machine configuration download using OAuth2 device flow.
The machine configuration is fetched from the URL specified with `talos.config` kernel argument, and by default this HTTP request is not authenticated.
When the OAuth2 authentication is enabled, Talos will authenticate the request using OAuth device flow first, and then pass the token to the machine configuration download endpoint.
## Prerequisites
Obtain the following information:
* OAuth client ID (mandatory)
* OAuth client secret (optional)
* OAuth device endpoint
* OAuth token endpoint
* OAuth scopes, audience (optional)
* extra Talos variables to send to the device auth endpoint (optional)
## Configuration
Set the following kernel parameters on the initial Talos boot to enable the OAuth flow:
* `talos.config` set to the URL of the machine configuration endpoint (which will be authenticated using OAuth)
* `talos.config.oauth.client_id` set to the OAuth client ID (required)
* `talos.config.oauth.client_secret` set to the OAuth client secret (optional)
* `talos.config.oauth.scope` set to the OAuth scopes (optional, repeat the parameter for multiple scopes)
* `talos.config.oauth.audience` set to the OAuth audience (optional)
* `talos.config.oauth.device_auth_url` set to the OAuth device endpoint (if not set defaults to `talos.config` URL with the path `/device/code`)
* `talos.config.oauth.token_url` set to the OAuth token endpoint (if not set defaults to `talos.config` URL with the path `/token`)
* `talos.config.oauth.extra_variable` set to the extra Talos variables to send to the device auth endpoint (optional, repeat the parameter for multiple variables)
The list of variables supported by the `talos.config.oauth.extra_variable` parameter is same as the [list of variables](../reference/kernel#talosconfig) supported by the `talos.config` parameter.
## Flow
On the initial Talos boot, when machine configuration is not available, Talos will print the following messages:
```text theme={null}
[talos] downloading config {"component": "controller-runtime", "controller": "config.AcquireController", "platform": "metal"}
[talos] waiting for network to be ready
[talos] [OAuth] starting the authentication device flow with the following settings:
[talos] [OAuth] - client ID: ""
[talos] [OAuth] - device auth URL: "https://oauth2.googleapis.com/device/code"
[talos] [OAuth] - token URL: "https://oauth2.googleapis.com/token"
[talos] [OAuth] - extra variables: ["uuid" "mac"]
[talos] waiting for variables: [uuid mac]
[talos] waiting for variables: [mac]
[talos] [OAuth] please visit the URL https://www.google.com/device and enter the code
[talos] [OAuth] waiting for the device to be authorized (expires at 14:46:55)...
```
If the OAuth service provides the complete verification URL, the QR code to scan is also printed to the console:
```text theme={null}
[talos] [OAuth] or scan the following QR code:
█████████████████████████████████
█████████████████████████████████
████ ▄▄▄▄▄ ██▄▀▀ ▀█ ▄▄▄▄▄ ████
████ █ █ █▄ ▀▄██▄██ █ █ ████
████ █▄▄▄█ ██▀▄██▄ ▀█ █▄▄▄█ ████
████▄▄▄▄▄▄▄█ ▀ █ ▀ █▄█▄▄▄▄▄▄▄████
████ ▀ ▄▄ ▄█ ██▄█ ███▄█▀████
████▀█▄ ▄▄▀▄▄█▀█▄██ ▄▀▄██▄ ▄████
████▄██▀█▄▄▄███▀ ▀█▄▄ ██ █▄ ████
████▄▀▄▄▄ ▄███ ▄ ▀ ▀▀▄▀▄▀█▄ ▄████
████▄█████▄█ █ ██ ▀ ▄▄▄ █▀▀████
████ ▄▄▄▄▄ █ █ ▀█▄█▄ █▄█ █▄ ████
████ █ █ █▄ ▄▀ ▀█▀▄▄▄ ▀█▄████
████ █▄▄▄█ █ ██▄ ▀ ▀███ ▀█▀▄████
████▄▄▄▄▄▄▄█▄▄█▄██▄▄▄▄█▄███▄▄████
█████████████████████████████████
```
Once the authentication flow is complete on the OAuth provider side, Talos will print the following message:
```text theme={null}
[talos] [OAuth] device authorized
[talos] fetching machine config from: "http://example.com/config.yaml"
[talos] machine config loaded successfully {"component": "controller-runtime", "controller": "config.AcquireController", "sources": ["metal"]}
```
# Role-based access control (RBAC)
Source: https://docs.siderolabs.com/talos/v1.13/security/rbac
Set up RBAC on the Talos Linux API.
Talos v0.11 introduced initial support for role-based access control (RBAC).
This guide will explain what that is and how to enable it without losing access to the cluster.
## RBAC in Talos
Talos uses certificates to authorize users.
The certificate subject's organization field is used to encode user roles.
There is a set of predefined roles that allow access to different [API methods](../reference/api):
* `os:admin` grants access to all methods;
* `os:operator` grants everything `os:reader` role does, plus additional methods: rebooting, shutting down, etcd backup, etcd alarm management, and so on;
* `os:reader` grants access to "safe" methods (for example, that includes the ability to list files, but does not include the ability to read files content);
* `os:etcd:backup` grants access to [`/machine.MachineService/EtcdSnapshot`](../reference/api#machine.EtcdSnapshotRequest) method.
Roles in the current `talosconfig` can be checked with the following command:
```sh theme={null}
$ talosctl config info
[...]
Roles: os:admin
[...]
```
RBAC is enabled by default in new clusters created with `talosctl` v0.11+ and disabled otherwise.
## Enabling RBAC
First, both the Talos cluster and `talosctl` tool should be [upgraded](../configure-your-talos-cluster/lifecycle-management/upgrading-talos).
Then the `talosctl config new` command should be used to generate a new client configuration with the `os:admin` role.
Additional configurations and certificates for different roles can be generated by passing `--roles` flag:
```sh theme={null}
talosctl config new --roles=os:reader reader
```
That command will create a new client configuration file `reader` with a new certificate with `os:reader` role.
After that, RBAC should be enabled in the machine configuration:
```yaml theme={null}
machine:
features:
rbac: true
```
# SELinux
Source: https://docs.siderolabs.com/talos/v1.13/security/selinux
SELinux security module support (experimental).
Talos Linux 1.10 introduces initial SELinux support.
Talos currently contains a basic SELinux policy that is designed to protect the OS from workloads, including privileged pods.
The policy denies access to machine configuration, prevents debuggers from attaching to system processes, however it is out of its scope to secure the Kubernetes components themselves.
## Configuration
SELinux is enabled by default in Talos 1.10 images.
The default mode is permissive, as currently some CNI and CSI solutions as well as extensions are incompatible with it.
For now, enforcing mode has only been tested with the Flannel CNI we ship by default.
These missing parts are being worked on to make SELinux available for more use cases.
### Disabling SELinux
On some occasions, you may want to disable SELinux completely, mostly if even permissive mode
is not compatible with some of your workloads or plugins.
SELinux also needs to be disabled if you are enabling AppArmor, because both security modules cannot be enabled at the same time.
If you want to disable SELinux, you can do so by adding `selinux=0` to the kernel cmdline.
This is most commonly done when creating the image via the configuration in the Image Factory.
### Mode of operation
You can query the SELinux state with:
```shell theme={null}
$ talosctl --nodes get SecurityState
NODE NAMESPACE TYPE ID VERSION SECUREBOOT UKISIGNINGKEYFINGERPRINT PCRSIGNINGKEYFINGERPRINT SELINUXSTATE
172.20.0.2 runtime SecurityState securitystate 1 false enabled, permissive
```
> Please note that SELinux is still in an experimental state in Talos Linux.
> Extensions currently do not support enforcing mode, which is a known missing feature being worked on.
> Expect some CNI and CSI plugins to not work in enforcing mode.
> Please report the issues you encounter with different configurations to help cover various usage scenarios.
> Enforcing mode should only be enabled on new installs as of version 1.10, since the upgrade path for enabling SELinux is still being worked on.
As for version 1.10, SELinux runs in permissive mode by default, which does not offer any extra protection, but allows to log denials.
SELinux can be put in enforcing mode (to actually prevent access when it is not authorized by the policy) by adding `enforcing=1` to the kernel cmdline.
This is most commonly done via the configuration in the Image Factory.
## Obtaining and processing denial logs
If SELinux has blocked some event from happening, it will log it to the audit log.
If the mode is permissive, the only implication of that would be a denial message, so permissive mode is useful for prototyping the policy.
You can check the logs with:
`talosctl --nodes logs auditd > audit.log`
You can get more insights on SELinux policy inner workings in the corresponding section of the [Developing Talos](../build-and-extend-talos/custom-images-and-development/developing-talos#selinux-policy-debugging-and-development) page.
# Source Talos Images
Source: https://docs.siderolabs.com/talos/v1.13/security/source-talos-images
Verifying source Talos container image signatures.
Sidero Labs signs the container images generated for the Talos release with [cosign](https://docs.sigstore.dev/cosign/overview/):
* `ghcr.io/siderolabs/installer` (Talos installer)
* `ghcr.io/siderolabs/talos` (Talos image for container runtime)
* `ghcr.io/siderolabs/talosctl` (`talosctl` client packaged as a container image)
* `ghcr.io/siderolabs/imager` (Talos install image generator)
* all [system extension images](https://github.com/siderolabs/extensions/)
## Verifying container image signatures
The `cosign` tool can be used to verify the signatures of the Talos container images:
```bash theme={null}
$ cosign verify --certificate-identity-regexp '(@siderolabs\.com$|^releasemgr-svc@talos-production\.iam\.gserviceaccount\.com$)' --certificate-oidc-issuer https://accounts.google.com ghcr.io/siderolabs/installer:v1.4.0
Verification for ghcr.io/siderolabs/installer:v1.4.0 --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- Existence of the claims in the transparency log was verified offline
- The code-signing certificate was verified using trusted certificate authority certificates
[{"critical":{"identity":{"docker-reference":"ghcr.io/siderolabs/installer"},"image":{"docker-manifest-digest":"sha256:f41795cc88f40eb1bc6b3c638c4a3123f6ef3c90627bfc35c04ebab82581e3ee"},"type":"cosign container image signature"},"optional":{"1.3.6.1.4.1.57264.1.1":"https://accounts.google.com","Bundle":{"SignedEntryTimestamp":"MEQCIERkQpgEnPWnfjUHIWO9QxC9Ute3/xJOc7TO5GUnu59xAiBKcFvrDWHoUYChT0/+gaazTrI+r0/GWSbi+Q+sEQ5AKA==","Payload":{"body":"eyJhcGlWZXJzaW9uIjoiMC4wLjEiLCJraW5kIjoiaGFzaGVkcmVrb3JkIiwic3BlYyI6eyJkYXRhIjp7Imhhc2giOnsiYWxnb3JpdGhtIjoic2hhMjU2IiwidmFsdWUiOiJkYjhjYWUyMDZmODE5MDlmZmI4NjE4ZjRkNjIzM2ZlYmM3NzY5MzliOGUxZmZkMTM1ODA4ZmZjNDgwNjYwNGExIn19LCJzaWduYXR1cmUiOnsiY29udGVudCI6Ik1FVUNJUURQWXhiVG5vSDhJTzBEakRGRE9rNU1HUjRjMXpWMys3YWFjczNHZ2J0TG1RSWdHczN4dVByWUgwQTAvM1BSZmZydDRYNS9nOUtzQVdwdG9JbE9wSDF0NllrPSIsInB1YmxpY0tleSI6eyJjb250ZW50IjoiTFMwdExTMUNSVWRKVGlCRFJWSlVTVVpKUTBGVVJTMHRMUzB0Q2sxSlNVTXhha05EUVd4NVowRjNTVUpCWjBsVlNIbEhaRTFQVEhkV09WbFFSbkJYUVRKb01qSjRVM1ZIZVZGM2QwTm5XVWxMYjFwSmVtb3dSVUYzVFhjS1RucEZWazFDVFVkQk1WVkZRMmhOVFdNeWJHNWpNMUoyWTIxVmRWcEhWakpOVWpSM1NFRlpSRlpSVVVSRmVGWjZZVmRrZW1SSE9YbGFVekZ3WW01U2JBcGpiVEZzV2tkc2FHUkhWWGRJYUdOT1RXcE5kMDVFUlRSTlZHZDZUbXBWTlZkb1kwNU5hazEzVGtSRk5FMVVaekJPYWxVMVYycEJRVTFHYTNkRmQxbElDa3R2V2tsNmFqQkRRVkZaU1V0dldrbDZhakJFUVZGalJGRm5RVVZaUVdKaVkwbDZUVzR3ZERBdlVEZHVUa0pNU0VscU1rbHlORTFQZGpoVVRrVjZUemNLUkVadVRXSldVbGc0TVdWdmExQnVZblJHTVZGMmRWQndTVm95VkV3NFFUUkdSMWw0YldFeGJFTk1kMkk0VEZOVWMzRlBRMEZZYzNkblowWXpUVUUwUndwQk1WVmtSSGRGUWk5M1VVVkJkMGxJWjBSQlZFSm5UbFpJVTFWRlJFUkJTMEpuWjNKQ1owVkdRbEZqUkVGNlFXUkNaMDVXU0ZFMFJVWm5VVlZqYWsweUNrbGpVa1lyTkhOVmRuRk5ia3hsU0ZGMVJIRkdRakZqZDBoM1dVUldVakJxUWtKbmQwWnZRVlV6T1ZCd2VqRlphMFZhWWpWeFRtcHdTMFpYYVhocE5Ga0tXa1E0ZDB0M1dVUldVakJTUVZGSUwwSkRSWGRJTkVWa1dWYzFhMk50VmpWTWJrNTBZVmhLZFdJeldrRmpNbXhyV2xoS2RtSkhSbWxqZVRWcVlqSXdkd3BMVVZsTFMzZFpRa0pCUjBSMmVrRkNRVkZSWW1GSVVqQmpTRTAyVEhrNWFGa3lUblprVnpVd1kzazFibUl5T1c1aVIxVjFXVEk1ZEUxRGMwZERhWE5IQ2tGUlVVSm5OemgzUVZGblJVaFJkMkpoU0ZJd1kwaE5Oa3g1T1doWk1rNTJaRmMxTUdONU5XNWlNamx1WWtkVmRWa3lPWFJOU1VkTFFtZHZja0puUlVVS1FXUmFOVUZuVVVOQ1NIZEZaV2RDTkVGSVdVRXpWREIzWVhOaVNFVlVTbXBIVWpSamJWZGpNMEZ4U2t0WWNtcGxVRXN6TDJnMGNIbG5Remh3TjI4MFFRcEJRVWRJYkdGbVp6Um5RVUZDUVUxQlVucENSa0ZwUVdKSE5tcDZiVUkyUkZCV1dUVXlWR1JhUmtzeGVUSkhZVk5wVW14c1IydHlSRlpRVXpsSmJGTktDblJSU1doQlR6WlZkbnBFYVVOYVFXOXZSU3RLZVdwaFpFdG5hV2xLT1RGS00yb3ZZek5CUTA5clJIcFhOamxaVUUxQmIwZERRM0ZIVTAwME9VSkJUVVFLUVRKblFVMUhWVU5OUVZCSlRUVjJVbVpIY0VGVWNqQTJVR1JDTURjeFpFOXlLMHhFSzFWQ04zbExUVWRMWW10a1UxTnJaMUp5U3l0bGNuZHdVREp6ZGdvd1NGRkdiM2h0WlRkM1NYaEJUM2htWkcxTWRIQnpjazFJZGs5cWFFSmFTMVoxVG14WmRXTkJaMVF4V1VWM1ZuZHNjR2QzYTFWUFdrWjRUemRrUnpONkNtVnZOWFJ3YVdoV1kyTndWMlozUFQwS0xTMHRMUzFGVGtRZ1EwVlNWRWxHU1VOQlZFVXRMUzB0TFFvPSJ9fX19","integratedTime":1681843022,"logIndex":18304044,"logID":"c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d"}},"Issuer":"https://accounts.google.com","Subject":"andrey.smirnov@siderolabs.com"}}]
```
The image should be signed using [cosign certificate authority flow](https://docs.sigstore.dev/certificate_authority/certificate-issuing-overview/) by a Sidero Labs employee with an email from `siderolabs.com` domain or by
a service account [releasemgr-svc@talos-production.iam.gserviceaccount.com](mailto:releasemgr-svc@talos-production.iam.gserviceaccount.com).
## Reproducible builds
Talos builds for `kernel`, `initramfs`, `talosctl`, ISO image, and container images are reproducible.
So you can verify that the build is the same as the one as provided on [GitHub releases page](https://github.com/siderolabs/talos/releases).
See [building Talos images](../build-and-extend-talos/custom-images-and-development/building-images) for more details.
# Verifying Image Signatures
Source: https://docs.siderolabs.com/talos/v1.13/security/verifying-image-signatures
Verifying container image signatures with Talos Linux.
Some components of Talos Linux are distributed as part of the Talos operating system image (for example, `containerd`), while
others are pulled as container images from the registry (for example, `ghcr.io/siderolabs/kubelet`).
All workloads running in Kubernetes clusters created with Talos Linux are also running as container images.
Thus, verifying the signatures of the container images is an important part of the security of the cluster, in order to prevent supply chain attacks and ensure that the images have not been tampered with.
Talos Linux provides a machine-wide configurable image signature verification policy which covers all image pulls on the machine,
including the ones initiated by the Talos operating system itself, as well as the ones initiated by Kubernetes for the workloads running in the cluster.
The image signature verification policy is a top-down list of rules which are evaluated in order.
The first rule that matches the image being pulled is applied, and the rest of the rules are ignored.
The rules can be configured to verify the signature, ignore the signature, or reject the image.
A rule matches on the image reference without the tag or digest, e.g. for the image `ghcr.io/siderolabs/installer:v1.13.0`, the reference matched is `ghcr.io/siderolabs/installer`.
The policy is configured using the [ImageVerificationConfig](../reference/configuration/security/imageverificationconfig) configuration document.
```yaml theme={null}
apiVersion: v1alpha1
kind: ImageVerificationConfig
rules:
- image: registry.k8s.io/*
keyless:
issuer: https://accounts.google.com
subject: krel-trust@k8s-releng-prod.iam.gserviceaccount.com
- image: ghcr.io/siderolabs/*
keyless:
issuer: https://accounts.google.com
subjectRegex: '(@siderolabs\.com$|^releasemgr-svc@talos-production\.iam\.gserviceaccount\.com$)'
- image: factory.talos.dev/*
keyless:
issuer: https://accounts.google.com
subject: image-factory-signing@talos-production.iam.gserviceaccount.com
```
In the example above, the first rule matches all images from `registry.k8s.io` and requires them to be signed by the Kubernetes release engineering team service account,
while the second rule matches all images from `ghcr.io/siderolabs` and requires them to be signed by a Sidero Labs employee or the Talos production release manager service account.
For the images from `factory.talos.dev` (Image Factory), the third rule requires them to be signed by the Talos production image factory service account.
Any other images that do not match any of the rules will be allowed by default.
This example set of rules can be used as a starting point for the image signature verification policy, and it can be customized to fit the specific needs of the cluster and the organization.
These rules cover all images used by Talos Linux itself by default, so they can be used as a base policy for the cluster and then additional rules can be added for the workloads running in the cluster.
It is also possible to create a closed policy that rejects all images that do not match any of the rules by adding a final rule with `deny: true`:
```yaml theme={null}
apiVersion: v1alpha1
kind: ImageVerificationConfig
rules:
- image: registry.k8s.io/*
keyless:
issuer: https://accounts.google.com
subject: krel-trust@k8s-releng-prod.iam.gserviceaccount.com
- image: "*"
deny: true
```
If some images are known to have no signatures, or if there are some images for which the signature verification is not required, a rule with `skip: true` can be added:
```yaml theme={null}
apiVersion: v1alpha1
kind: ImageVerificationConfig
rules:
- image: registry.k8s.io/e2e-test-images/*
skip: true # skip verification for e2e test images since they are not signed
- image: registry.k8s.io/*
keyless:
issuer: https://accounts.google.com
subject: krel-trust@k8s-releng-prod.iam.gserviceaccount.com
```
For the rules which verify the signature, there are two verification mechanisms supported: keyless verification based on issuer/subject claims, and verification based on public keys.
Both mechanisms are based on the [cosign](https://docs.sigstore.dev/) tool.
Keyless verification is more convenient since it does not require managing the public keys, but it requires Talos machines to have access to the TUF repository with the trusted root certificates.
Both Kubernetes and Sidero Labs use keyless signing methods.
> Note: Keyless verification will not work on air-gapped clusters that do not have access to the Internet.
Public key verification does not have the above limitation, but it requires generating and managing the signing keys and making the public keys available to the Talos machines.
The image verification policy is applied at the moment of pulling the image, so if the image is already present in the local image store, the signature will not be verified again.
The signature verification can be verified by checking the image labels, for the presence of the `talos.dev/verified` label:
```bash theme={null}
$ talosctl image ls
NODE IMAGE DIGEST SIZE LABELS CREATED
172.20.0.2 registry.k8s.io/kube-apiserver:v1.36.0-beta.0 sha256:8738e8e72f834e981f7e9bf0d65bb2bafb3c5cac6404bbeca2fb7dc4461eb45e 30 MB io.cri-containerd.image=managed,talos.dev/verified=verified via legacy signature (bundle verified true) 2026-03-27T16:18:32Z
172.20.0.2 registry.k8s.io/kube-apiserver@sha256:8738e8e72f834e981f7e9bf0d65bb2bafb3c5cac6404bbeca2fb7dc4461eb45e sha256:8738e8e72f834e981f7e9bf0d65bb2bafb3c5cac6404bbeca2fb7dc4461eb45e 30 MB io.cri-containerd.image=managed,talos.dev/verified=verified via legacy signature (bundle verified true) 2026-03-27T16:18:32Z
```
And same command for images pulled by Talos:
```bash theme={null}
talosctl image ls --namespace system
NODE IMAGE DIGEST SIZE LABELS CREATED
172.20.0.2 ghcr.io/siderolabs/kubelet:v1.36.0-beta.0 sha256:9351f21d2dd799486132d7d5d2824d335463710802dee813e18a452e375f48d5 64 MB talos.dev/verified=verified via bundle 2026-03-27T16:18:04Z
172.20.0.2 ghcr.io/siderolabs/kubelet@sha256:9351f21d2dd799486132d7d5d2824d335463710802dee813e18a452e375f48d5 sha256:9351f21d2dd799486132d7d5d2824d335463710802dee813e18a452e375f48d5 64 MB talos.dev/verified=verified via bundle 2026-03-27T16:18:04Z
```
# SBOMs
Source: https://docs.siderolabs.com/talos/v1.13/advanced-guides/SBOM
A guide on using Software Bill of Materials for Talos Linux.
Software Bill of Materials (SBOM) is a formal record containing the details and supply chain relationships of various components used in building a software product.
SBOMs are used to provide transparency and traceability of software components, which is essential for security, compliance, and efficient management of software supply chains.
Talos Linux provides SBOMs for core operating system components, including the Linux kernel, built-in components like `containerd`, and other software packages used to build Talos Linux.
When a system extension is installed, it can also provide its own SBOM, which will be included in the overall SBOM for the Talos Linux system.
## Acquiring SBOMs
SBOMs for Talos Linux are provided in SPDX format, which is a standard format for representing SBOMs.
You can acquire SBOMs for Talos Linux in the following ways:
* Download the SBOM for a specific Talos Linux release from the GitHub release page:
* `talos-amd64.spdx.json` for the amd64 architecture.
* `talos-arm64.spdx.json` for the arm64 architecture.
* Acquire the SBOM from a running Talos Linux system using the `talosctl` command:
* core Talos Linux SBOM in the `/usr/share/spdx` directory.
* extension SBOMs in the `/usr/local/share/spdx` directory.
## SBOMs as resources
Talos Linux SBOMs are also available as resources in the Talos Linux system.
You can access the SBOMs using the `talosctl` command:
You can also get the SBOM for a specific component using the `talosctl get sbom` command:
```yaml theme={null}
# talosctl get sbom kernel -o yaml
node: 172.20.0.2
metadata:
namespace: runtime
type: SBOMItems.talos.dev
id: kernel
version: 1
owner: runtime.SBOMItemController
phase: running
created: 2025-07-24T14:20:29Z
updated: 2025-07-24T14:20:29Z
spec:
name: kernel
version: 6.12.38
license: GPL-2.0-only
cpes:
- cpe:2.3:o:linux:linux_kernel:6.12.38:*:*:*:*:*:*:*
```
## Scanning SBOMs
You can scan SBOMs for known vulnerabilities using tools like [Grype](https://github.com/anchore/grype).
You will need two source files for scanning:
* The SBOM file in SPDX format.
* The vulnerability exclusion database (VEX).
VEX database is used to filter out vulnerabilities that are not applicable to the specific software version or configuration,
which helps to reduce false positives in vulnerability scanning.
The VEX database is available to Enterprise customers of Talos Linux. Contact [Sidero support](https://www.siderolabs.com/contact/) for access.
The basic command to scan the SBOM is as follows:
```bash theme={null}
grype sbom:talos-amd64.spdx.json
```
With VEX database, the command becomes:
```bash theme={null}
grype sbom:talos-amd64.spdx.json --vex vex.json
```
# Install KubeVirt on Talos
Source: https://docs.siderolabs.com/talos/v1.13/advanced-guides/install-kubevirt
This is a guide on how to get started with KubeVirt on Talos
KubeVirt allows you to run virtual machines on Kubernetes.
It runs with QEMU and KVM to provide a seamless virtual machine experience and can be mixed with containerized workloads.
This guide explains on how to install KubeVirt on Talos.
## Prerequisites
For KubeVirt and Talos to work you have to enable certain configurations in the BIOS and configure Talos properly for it to work.
### Enable virtualization in your BIOS
On many new PCs and servers, virtualization is enabled by default.
Please consult your manufacturer on how to enable this in the BIOS.
You can also run KubeVirt from within a virtual machine.
For that to work you have to enable Nested Virtualization.
This can also be done in the BIOS.
### Configure your network interface in bridge mode (optional)
When you want to leverage [Multus](../../../kubernetes-guides/cni/multus) to give your virtual machines direct access to your node network, your bridge needs to be configured properly.
This can be done by setting your network interface in bridge mode.
You can look up the network interface name by using the following command:
```bash theme={null}
$ talosctl get links -n 10.99.101.9
NODE NAMESPACE TYPE ID VERSION TYPE KIND HW ADDR OPER STATE LINK STATE
10.99.101.9 network LinkStatus bond0 1 ether bond 52:62:01:53:5b:a7 down false
10.99.101.9 network LinkStatus br0 3 ether bridge bc:24:11:a1:98:fc up true
10.99.101.9 network LinkStatus cni0 9 ether bridge 1e:5e:99:8f:1e:19 up true
10.99.101.9 network LinkStatus dummy0 1 ether dummy 62:1c:3e:d5:72:11 down false
10.99.101.9 network LinkStatus eth0 5 ether bc:24:11:a1:98:fc
```
In this case, this network interface is called `eth0`.
Now you can configure your bridge properly.
This can be done in the machine config of your node:
```yaml theme={null}
machine:
interfaces:
- interface: br0
addresses:
- 10.99.101.9/24
bridge:
stp:
enabled: true
interfaces:
- eth0 # This must be changed to your matching interface name
routes:
- network: 0.0.0.0/0 # The route's network (destination).
gateway: 10.99.101.254 # The route's gateway (if empty, creates link scope route).
metric: 1024 # The optional metric for the route.
```
### Install the `local-path-provisioner`
When we are using KubeVirt, we are also installing the CDI (containerized data importer) operator.
For this to work properly, we have to install the `local-path-provisioner`.
This CSI can be used to write scratch space when importing images with the CDI.
You can install the `local-path-provisioner` by following [this guide](../../../kubernetes-guides/csi/local-storage).
### Configure storage
If you would like to use features such as `LiveMigration` shared storage is necessary.
You can either choose to install a CSI that connects to NFS or you can install Longhorn, for example.
For more information on how to install Longhorn on Talos you can follow [this](https://longhorn.io/docs/1.7.2/advanced-resources/os-distro-specific/talos-linux-support/) link.
To install the NFS-CSI driver, you can follow [This](https://github.com/kubernetes-csi/csi-driver-nfs/blob/master/docs/install-csi-driver-v4.9.0.md) guide.
After the installation of the NFS-CSI driver is done, you can create a storage class for the NFS CSI driver to work:
```yaml theme={null}
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-csi
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: nfs.csi.k8s.io
parameters:
server: 10.99.102.253
share: /mnt/data/nfs/kubernetes_csi
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
- nfsvers=3
- nolock
```
Note that this is just an example.
Make sure to set the `nolock` option.
If not, the nfs-csi storageclass won't work, because talos doesn't have a `rpc.statd` daemon running.
### Install `virtctl`
`virtctl` is needed for communication between the CLI and the KubeVirt api server.
You can install the `virtctl` client directly by running:
```bash theme={null}
export VERSION=$(curl https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt)
wget https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/virtctl-${VERSION}-linux-amd64
```
Or you can use [krew](https://github.com/kubernetes-sigs/krew/#installation) to integrate it nicely in `kubectl`:
```bash theme={null}
kubectl krew install virt
```
## Installing KubeVirt
After the necessary preparations are done, you can now install KubeVirt.
This can either be done through the [Operator Lifecycle Manager](https://olm.operatorframework.io/docs/getting-started/) or by just simply applying a YAML file.
We will keep this simple and do the following:
```bash theme={null}
# Point at latest release
export RELEASE=$(curl https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt)
# Deploy the KubeVirt operator
kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml
```
After the operator is installed, it is time to apply the Custom Resource (CR) for the operator to fully deploy KubeVirt.
```yaml theme={null}
---
apiVersion: kubevirt.io/v1
kind: KubeVirt
metadata:
name: kubevirt
namespace: kubevirt
spec:
configuration:
developerConfiguration:
featureGates:
- LiveMigration
- NetworkBindingPlugins
smbios:
sku: "TalosCloud"
version: "v0.1.0"
manufacturer: "Talos Virtualization"
product: "talosvm"
family: "ccio"
workloadUpdateStrategy:
workloadUpdateMethods:
- LiveMigrate # enable if you have deployed either Longhorn or NFS-CSI for shared storage.
```
### KubeVirt configuration options
In this yaml file we specified certain configurations:
#### `featureGates`
KubeVirt has a set of features that are not mature enough to be enabled by default.
As such, they are protected by a Kubernetes concept called feature gates.
More information about the feature gates can be found in the [KubeVirt](https://kubevirt.io/user-guide/cluster_admin/activating_feature_gates/) documentation.
In this example we enable:
* `LiveMigration` -- For live migration of virtual machines to other nodes
* `NetworkBindingPlugins` -- This is needed for Multus to work.
#### `smbios`
Here we configure a specific smbios configuration.
This can be useful when you want to give your virtual machines a own sku, manufacturer name etc.
#### `workloadUpdateStrategy`
If this is configured, virtual machines will be live migrated to other nodes when KubeVirt is updated.
## Installing CDI
The CDI (containerized data importer) is needed to import virtual disk images in your KubeVirt cluster.
The CDI can do the following:
* Import images of type:
* qcow2
* raw
* iso
* Import disks from either:
* http/https
* uploaded through virtctl
* Container registry
* Another PVC
You can either import these images by creating a DataVolume CR or by integrating this in your `VirtualMachine` CR.
When applying either the `DataVolume` CR or the `VirtualMachine` CR with a `dataVolumeTemplates`, the CDI kicks in and will do the following:
* creates a PVC with the requirements from either the `DataVolume` or the `dataVolumeTemplates`
* starts a pod
* writes temporary scratch space to local disk
* downloads the image
* extracts it to the temporary scratch space
* copies the image to the PVC
Installing the CDI is very simple:
```bash theme={null}
# Point to latest release
export TAG=$(curl -s -w %{redirect_url} \
https://github.com/kubevirt/containerized-data-importer/releases/latest)
export VERSION=$(echo ${TAG##*/})
# install operator
kubectl create -f \
https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml
```
After that, you can apply a CDI CR for the CDI operator to fully deploy CDI:
```yaml theme={null}
apiVersion: cdi.kubevirt.io/v1beta1
kind: CDI
metadata:
name: cdi
spec:
config:
scratchSpaceStorageClass: local-path
podResourceRequirements:
requests:
cpu: "100m"
memory: "60M"
limits:
cpu: "750m"
memory: "2Gi"
```
This CR has some special settings that are needed for CDI to work properly:
### `scratchSpaceStorageClass`
This is the storage class that we installed earlier with the `local-path-provisioner`.
This is needed for the CDI to write scratch space to local disk before importing the image
### `podResourceRequirements`
In many cases the default resource requests and limits are not sufficient for the importer pod to import the image.
This will result in a crash of the importer pod.
After applying this yaml file, the CDI operator is ready.
## Creating your first virtual machine
Now it is time to create your first virtual machine in KubeVirt.
Below we will describe two examples:
* A virtual machine with the default CNI
* A virtual machine with Multus
### Basic virtual machine example with default CNI
```yaml theme={null}
---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: fedora-vm
spec:
running: false
template:
metadata:
labels:
kubevirt.io/vm: fedora-vm
annotations:
kubevirt.io/allow-pod-bridge-network-live-migration: "true"
spec:
evictionStrategy: LiveMigrate
domain:
cpu:
cores: 2
resources:
requests:
memory: 4G
devices:
disks:
- name: fedora-vm-pvc
disk:
bus: virtio
- name: cloudinitdisk
disk:
bus: virtio
interfaces:
- name: podnet
masquerade: {}
networks:
- name: podnet
pod: {}
volumes:
- name: fedora-vm-pvc
persistentVolumeClaim:
claimName: fedora-vm-pvc
- name: cloudinitdisk
cloudInitNoCloud:
networkData: |
network:
version: 1
config:
- type: physical
name: eth0
subnets:
- type: dhcp
userData: |-
#cloud-config
users:
- name: cloud-user
ssh_authorized_keys:
- ssh-rsa ....
sudo: ['ALL=(ALL) NOPASSWD:ALL']
groups: sudo
shell: /bin/bash
runcmd:
- "sudo touch /root/installed"
- "sudo dnf update"
- "sudo dnf install httpd fastfetch -y"
- "sudo systemctl daemon-reload"
- "sudo systemctl enable httpd"
- "sudo systemctl start --no-block httpd"
dataVolumeTemplates:
- metadata:
name: fedora-vm-pvc
spec:
storage:
resources:
requests:
storage: 35Gi
accessModes:
- ReadWriteMany
storageClassName: "nfs-csi"
source:
http:
url: "https://fedora.mirror.wearetriple.com/linux/releases/40/Cloud/x86_64/images/Fedora-Cloud-Base-Generic.x86_64-40-1.14.qcow2"
```
In this examples we install a basic Fedora 40 virtual machine and install a webserver.
After applying this YAML, the CDI will import the image and create a `Datavolume`.
You can monitor this process by running:
```bash theme={null}
kubectl get dv -w
```
After the `DataVolume` is created, you can start the virtual machine:
```bash theme={null}
kubectl virt start fedora-vm
```
By starting the virtual machine, KubeVirt will create a instance of that `VirtualMachine` called `VirtualMachineInstance`:
```bash theme={null}
kubectl get virtualmachineinstance
NAME AGE PHASE IP NODENAME READY
fedora-vm 13s Running 10.244.4.92 kube1 True
```
You can view the console of the virtual machine by running:
```bash theme={null}
kubectl virt console fedora-vm
```
or by running:
```bash theme={null}
kubectl virt vnc fedora-vm
```
with the `console` command it will open a terminal to the virtual machine.
With the `vnc` command, it will open `vncviewer`.
Note that a `vncviewer` needs to installed for it to work.
Now you can create a `Service` object to expose the virtual machine to the outside.
In this example we will use [MetalLB](https://metallb.universe.tf/) as a LoadBalancer.
```yaml theme={null}
apiVersion: v1
kind: Service
metadata:
labels:
kubevirt.io/vm: fedora-vm
name: fedora-vm
spec:
ipFamilyPolicy: PreferDualStack
externalTrafficPolicy: Local
ports:
- name: ssh
port: 22
protocol: TCP
targetPort: 22
- name: httpd
port: 80
protocol: TCP
targetPort: 80
selector:
kubevirt.io/vm: fedora-vm
type: LoadBalancer
```
```bash theme={null}
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
fedora-vm LoadBalancer 10.96.14.253 10.99.50.1 22:31149/TCP,80:31445/TCP 2s
```
And we can reach the server with either ssh or http:
```bash theme={null}
$ nc -zv 10.99.50.1 22
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 10.99.50.1:22.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
$ nc -zv 10.99.50.1 80
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 10.99.50.1:80.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
```
### Basic virtual machine example with Multus
```yaml theme={null}
---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: fedora-vm
spec:
running: false
template:
metadata:
labels:
kubevirt.io/vm: fedora-vm
annotations:
kubevirt.io/allow-pod-bridge-network-live-migration: "true"
spec:
evictionStrategy: LiveMigrate
domain:
cpu:
cores: 2
resources:
requests:
memory: 4G
devices:
disks:
- name: fedora-vm-pvc
disk:
bus: virtio
- name: cloudinitdisk
disk:
bus: virtio
interfaces:
- name: external
bridge: {} # We use the bridge interface.
networks:
- name: external
multus:
networkName: namespace/networkattachmentdefinition # This is the NetworkAttachmentDefinition. See multus docs for more info.
volumes:
- name: fedora-vm-pvc
persistentVolumeClaim:
claimName: fedora-vm-pvc
- name: cloudinitdisk
cloudInitNoCloud:
networkData: |
network:
version: 1
config:
- type: physical
name: eth0
subnets:
- type: dhcp
userData: |-
#cloud-config
users:
- name: cloud-user
ssh_authorized_keys:
- ssh-rsa ....
sudo: ['ALL=(ALL) NOPASSWD:ALL']
groups: sudo
shell: /bin/bash
runcmd:
- "sudo touch /root/installed"
- "sudo dnf update"
- "sudo dnf install httpd fastfetch -y"
- "sudo systemctl daemon-reload"
- "sudo systemctl enable httpd"
- "sudo systemctl start --no-block httpd"
dataVolumeTemplates:
- metadata:
name: fedora-vm-pvc
spec:
storage:
resources:
requests:
storage: 35Gi
accessModes:
- ReadWriteMany
storageClassName: "nfs-csi"
source:
http:
url: "https://fedora.mirror.wearetriple.com/linux/releases/40/Cloud/x86_64/images/Fedora-Cloud-Base-Generic.x86_64-40-1.14.qcow2"
```
In this example we will create a virtual machine that is bound to the bridge interface with the help of [Multus](../../../kubernetes-guides/cni/multus).
You can start the machine with `kubectl virt start fedora-vm`.
After that you can look up the ip address of the virtual machine with
```bash theme={null}
kubectl get vmi -owide
NAME AGE PHASE IP NODENAME READY LIVE-MIGRATABLE PAUSED
fedora-vm 6d9h Running 10.99.101.53 kube1 True True
```
## Other forms of management
There is a project called [KubeVirt-Manager](https://kubevirt-manager.io/) for managing virtual machines with KubeVirt through a nice web interface.
You can also choose to deploy virtual machines with ArgoCD or Flux.
## Documentation
KubeVirt has a huge documentation page where you can check out everything on running virtual machines with KubeVirt.
The documentation can be found [here](https://kubevirt.io/user-guide/).
# Migrating from Kubeadm
Source: https://docs.siderolabs.com/talos/v1.13/advanced-guides/migrating-from-kubeadm
Transition an existing kubeadm cluster to Talos with a controlled, node-by-node migration process.
It is possible to migrate Talos from a cluster that is created using
[kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/) to Talos.
High-level steps are the following:
1. Collect CA certificates and a bootstrap token from a control plane node.
2. Create a Talos machine config with the CA certificates with the ones you collected.
3. Update control plane endpoint in the machine config to point to the existing control plane (i.e. your load balancer address).
4. Boot a new Talos machine and apply the machine config.
5. Verify that the new control plane node is ready.
6. Remove one of the old control plane nodes.
7. Repeat the same steps for all control plane nodes.
8. Verify that all control plane nodes are ready.
9. Repeat the same steps for all worker nodes, using the machine config generated for the workers.
## Remarks on kube-apiserver load balancer
While migrating to Talos, you need to make sure that your kube-apiserver load balancer is in place
and keeps pointing to the correct set of control plane nodes.
This process depends on your load balancer setup.
If you are using an LB that is external to the control plane nodes (e.g. cloud provider LB, F5 BIG-IP, etc.),
you need to make sure that you update the backend IPs of the load balancer to point to the control plane nodes as
you add Talos nodes and remove kubeadm-based ones.
If your load balancing is done on the control plane nodes (e.g. keepalived + haproxy on the control plane nodes),
you can do the following:
1. Add Talos nodes and remove kubeadm-based ones while updating the haproxy backends
to point to the newly added nodes except the last kubeadm-based control plane node.
2. Turn off keepalived to drop the virtual IP used by the kubeadm-based nodes (introduces kube-apiserver downtime).
3. Set up a virtual-IP based new load balancer on the new set of Talos control plane nodes.
Use the previous LB IP as the LB virtual IP.
4. Verify apiserver connectivity over the Talos-managed virtual IP.
5. Migrate the last control-plane node.
## Prerequisites
* Admin access to the kubeadm-based cluster
* Access to the `/etc/kubernetes/pki` directory (e.g. SSH & root permissions)
on the control plane nodes of the kubeadm-based cluster
* Access to kube-apiserver load-balancer configuration
## Step-by-step guide
1. Download `/etc/kubernetes/pki` directory from a control plane node of the kubeadm-based cluster.
2. Create a new join token for the new control plane nodes:
```bash theme={null}
# inside a control plane node
kubeadm token create --ttl 0
```
3. Create Talos secrets from the PKI directory you downloaded on step 1 and the token you generated on step 2:
```bash theme={null}
talosctl gen secrets --kubernetes-bootstrap-token --from-kubernetes-pki
```
4. Create a new Talos config from the secrets:
```bash theme={null}
talosctl gen config --with-secrets secrets.yaml https://
```
5. Collect the information about the kubeadm-based cluster from the kubeadm configmap:
```bash theme={null}
kubectl get configmap -n kube-system kubeadm-config -oyaml
```
Take note of the following information in the `ClusterConfiguration`:
* `.controlPlaneEndpoint`
* `.networking.dnsDomain`
* `.networking.podSubnet`
* `.networking.serviceSubnet`
6. Replace the following information in the generated `controlplane.yaml`:
* `.cluster.network.cni.name` with `none`
* `.cluster.network.podSubnets[0]` with the value of the `networking.podSubnet` from the previous step
* `.cluster.network.serviceSubnets[0]` with the value of the `networking.serviceSubnet` from the previous step
* `.cluster.network.dnsDomain` with the value of the `networking.dnsDomain` from the previous step
7. Go through the rest of `controlplane.yaml` and `worker.yaml` to customize them according to your needs, especially :
* `.cluster.secretboxEncryptionSecret` should be either removed if you don't currently use `EncryptionConfig` on your `kube-apiserver` or set to the correct value
8. Make sure that, on your current Kubeadm cluster, the first `--service-account-issuer=` parameter in `/etc/kubernetes/manifests/kube-apiserver.yaml` is equal to the value of `.cluster.controlPlane.endpoint` in `controlplane.yaml`.
If it's not, add a new `--service-account-issuer=` parameter with the correct value before your current one in `/etc/kubernetes/manifests/kube-apiserver.yaml` on all of your control planes nodes, and restart the kube-apiserver containers.
9. Bring up a Talos node to be the initial Talos control plane node.
10. Apply the generated `controlplane.yaml` to the Talos control plane node:
```bash theme={null}
talosctl --nodes apply-config --insecure --file controlplane.yaml
```
11. Wait until the new control plane node joins the cluster and is ready.
```bash theme={null}
kubectl get node -owide --watch
```
12. Update your load balancer to point to the new control plane node.
13. Drain the old control plane node you are replacing:
```bash theme={null}
kubectl drain --delete-emptydir-data --force --ignore-daemonsets --timeout=10m
```
14. Remove the old control plane node from the cluster:
```bash theme={null}
kubectl delete node