> ## Documentation Index > Fetch the complete documentation index at: https://docs.siderolabs.com/llms.txt > Use this file to discover all available pages before exploring further. # CA Rotation > How to rotate Talos and Kubernetes API root certificate authorities. export const release_v1_13 = 'v1.13.1'; export const VersionWarningBanner = () => { const latestVersion = "v1.13"; const [latestUrl, setLatestUrl] = useState(null); const [currentVersion, setCurrentVersion] = useState(null); const [isBeta, setIsBeta] = useState(false); const parseVersion = v => v.replace("v", "").split(".").map(Number); const isGreaterVersion = (a, b) => { const [aMajor, aMinor] = parseVersion(a); const [bMajor, bMinor] = parseVersion(b); if (aMajor > bMajor) return true; if (aMajor === bMajor && aMinor > bMinor) return true; return false; }; useEffect(() => { if (typeof window === "undefined") return; const {pathname, hash, search} = window.location; const match = pathname.match(/\/talos\/(v\d+\.\d+)\//); if (!match) return; const detectedVersion = match[1]; if (detectedVersion === latestVersion) return; setCurrentVersion(detectedVersion); if (isGreaterVersion(detectedVersion, latestVersion)) { setIsBeta(true); } const newPath = pathname.replace(`/talos/${detectedVersion}/`, `/talos/${latestVersion}/`); setLatestUrl(`${newPath}${search}${hash}`); }, []); if (!latestUrl || !currentVersion) return null; return
{isBeta ? <> ⚠️ You are viewing a beta version of Talos ({currentVersion}). This version may be unstable. View latest stable version {latestVersion} → : <> ⚠️ You are viewing an older version of Talos ({currentVersion}). View the latest version {latestVersion} → }
; }; In general, you almost never need to rotate the root CA certificate and key for the Talos API and Kubernetes API. Talos sets up root certificate authorities with the lifetime of 10 years, and all Talos and Kubernetes API certificates are issued by these root CAs. So the rotation of the root CA is only needed if: * you suspect that the private key has been compromised; * you want to revoke access to the cluster for a leaked `talosconfig` or `kubeconfig`; * once in 10 years. ## Overview There are some details which make Talos and Kubernetes API root CA rotation a bit different, but the general flow is the same: * generate new CA certificate and key; * add new CA certificate as 'accepted', so new certificates will be accepted as valid; * swap issuing CA to the new one, old CA as accepted; * refresh all certificates in the cluster; * remove old CA from 'accepted'. At the end of the flow, old CA is completely removed from the cluster, so all certificates issued by it will be considered invalid. Both rotation flows are described in detail below. ## Talos API ### Automated Talos API CA rotation Talos API CA rotation doesn't interrupt connections within the cluster, and it doesn't require a reboot of the nodes. Run the following command in dry-run mode to see the steps which will be taken: ```shell theme={null} $ talosctl -n rotate-ca --dry-run=true --talos=true --kubernetes=false > Starting Talos API PKI rotation, dry-run mode true... > Using config context: "talos-default" > Using Talos API endpoints: ["172.20.0.2"] > Cluster topology: - control plane nodes: ["172.20.0.2"] - worker nodes: ["172.20.0.3"] > Current Talos CA: ... ``` No changes will be done to the cluster in dry-run mode, so you can safely run it to see the steps. Before proceeding, make sure that you can capture the output of `talosctl` command, as it will contain the new CA certificate and key. Record a list of Talos API users to make sure they can all be updated with new `talosconfig`. Run the following command to rotate the Talos API CA: {` $ talosctl -n rotate-ca --dry-run=false --talos=true --kubernetes=false > Starting Talos API PKI rotation, dry-run mode false... > Using config context: "talos-default-268" > Using Talos API endpoints: ["172.20.0.2"] > Cluster topology: - control plane nodes: ["172.20.0.2"] - worker nodes: ["172.20.0.3"] > Current Talos CA: ... > New Talos CA: ... > Generating new talosconfig: context: talos-default contexts: talos-default: .... > Verifying connectivity with existing PKI: - 172.20.0.2: OK (version ${release_v1_13}) - 172.20.0.3: OK (version ${release_v1_13}) > Adding new Talos CA as accepted... - 172.20.0.2: OK - 172.20.0.3: OK > Verifying connectivity with new client cert, but old server CA: 2024/04/17 21:26:07 retrying error: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: remote error: tls: unknown certificate authority" - 172.20.0.2: OK (version ${release_v1_13}) - 172.20.0.3: OK (version ${release_v1_13}) > Making new Talos CA the issuing CA, old Talos CA the accepted CA... - 172.20.0.2: OK - 172.20.0.3: OK > Verifying connectivity with new PKI: 2024/04/17 21:26:08 retrying error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"x509: Ed25519 verification failure\" while trying to verify candidate authority certificate \"talos\")" - 172.20.0.2: OK (version ${release_v1_13}) - 172.20.0.3: OK (version ${release_v1_13}) > Removing old Talos CA from the accepted CAs... - 172.20.0.2: OK - 172.20.0.3: OK > Verifying connectivity with new PKI: - 172.20.0.2: OK (version ${release_v1_13}) - 172.20.0.3: OK (version ${release_v1_13}) > Writing new talosconfig to "talosconfig" `} Once the rotation is done, stash the new Talos CA, update `secrets.yaml` (if using that for machine configuration generation) with new CA key and certificate. The new client `talosconfig` is written to the current directory as `talosconfig`. You can merge it to the default location with `talosctl config merge ./talosconfig`. If other client access `talosconfig` files needs to be generated, use `talosctl config new` with new `talosconfig`. > Note: if using [Talos API access from Kubernetes](../../../kubernetes-guides/advanced-guides/talos-api-access-from-k8s) feature, pods might need to be restarted manually to pick up new `talosconfig`. ### Manual steps for Talos API CA rotation 1. Generate new Talos CA (e.g. use `talosctl gen secrets` and use Talos CA). 2. Patch machine configuration on all nodes updating `.machine.acceptedCAs` with new CA certificate. 3. Generate `talosconfig` with client certificate generated with new CA, but still using old CA as server CA, verify connectivity, Talos should accept new client certificate. 4. Patch machine configuration on all nodes updating `.machine.ca` with new CA certificate and key, and keeping old CA certificate in `.machine.acceptedCAs` (on worker nodes `.machine.ca` doesn't have the key). 5. Generate `talosconfig` with both client certificate and server CA using new CA PKI, verify connectivity. 6. Remove old CA certificate from `.machine.acceptedCAs` on all nodes. 7. Verify connectivity. ## Kubernetes API ### Automated Kubernetes API CA rotation The automated process only rotates Kubernetes API CA, used by the `kube-apiserver`, `kubelet`, etc. Other Kubernetes secrets might need to be rotated manually as required. Kubernetes pods might need to be restarted to handle changes, and communication within the cluster might be disrupted during the rotation process. Run the following command in dry-run mode to see the steps which will be taken: ```shell theme={null} $ talosctl -n rotate-ca --dry-run=true --talos=false --kubernetes=true > Starting Kubernetes API PKI rotation, dry-run mode true... > Cluster topology: - control plane nodes: ["172.20.0.2"] - worker nodes: ["172.20.0.3"] > Building current Kubernetes client... > Current Kubernetes CA: ... ``` Before proceeding, make sure that you can capture the output of `talosctl` command, as it will contain the new CA certificate and key. As Talos API access will not be disrupted, the changes can be reverted back if needed by reverting machine configuration. Run the following command to rotate the Kubernetes API CA: ```shell theme={null} $ talosctl -n rotate-ca --dry-run=false --talos=false --kubernetes=true > Starting Kubernetes API PKI rotation, dry-run mode false... > Cluster topology: - control plane nodes: ["172.20.0.2"] - worker nodes: ["172.20.0.3"] > Building current Kubernetes client... > Current Kubernetes CA: ... > New Kubernetes CA: ... > Verifying connectivity with existing PKI... - OK (2 nodes ready) > Adding new Kubernetes CA as accepted... - 172.20.0.2: OK - 172.20.0.3: OK > Making new Kubernetes CA the issuing CA, old Kubernetes CA the accepted CA... - 172.20.0.2: OK - 172.20.0.3: OK > Building new Kubernetes client... > Verifying connectivity with new PKI... 2024/04/17 21:45:52 retrying error: Get "https://172.20.0.1:6443/api/v1/nodes": EOF - OK (2 nodes ready) > Removing old Kubernetes CA from the accepted CAs... - 172.20.0.2: OK - 172.20.0.3: OK > Verifying connectivity with new PKI... - OK (2 nodes ready) > Kubernetes CA rotation done, new 'kubeconfig' can be fetched with `talosctl kubeconfig`. ``` At the end of the process, Kubernetes control plane components will be restarted to pick up CA certificate changes. Each node `kubelet` will re-join the cluster with new client certificate. New `kubeconfig` can be fetched with `talosctl kubeconfig` command from the cluster. Kubernetes pods might need to be restarted manually to pick up changes to the Kubernetes API CA. ### Manual steps for Kubernetes API CA rotation Steps are similar [to the Talos API CA rotation](#manual-steps-for-talos-api-ca-rotation), but use: * `.cluster.acceptedCAs` in place of `.machine.acceptedCAs`; * `.cluster.ca` in place of `.machine.ca`; * `kubeconfig` in place of `talosconfig`.