> ## Documentation Index
> Fetch the complete documentation index at: https://docs.siderolabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploy Cilium CNI

> In this guide you will learn how to set up Cilium CNI on Talos.

export const k8s_release = '1.36.0';

> Cilium can be installed either via the `cilium` cli or using `helm`.

This documentation will outline installing Cilium CNI v1.18.0 on Talos in six different ways.
Adhering to Talos principles we'll deploy Cilium with IPAM mode set to Kubernetes, and using the `cgroupv2` and `bpffs` mount that talos already provides.
As Talos does not allow loading kernel modules by Kubernetes workloads, `SYS_MODULE` capability needs to be dropped from the Cilium default set of values, this override can be seen in the helm/cilium cli install commands.
Each method can either install Cilium using kube proxy (default) or without: [Kubernetes Without kube-proxy](https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/).

In this guide we assume that [KubePrism](../advanced-guides/kubeprism) is enabled and configured to use the port 7445.

## Machine configuration preparation

When generating the machine config for a node set the CNI to none.
For example using a config patch:

<Tabs>
  <Tab title="With kube-proxy">
    ```bash theme={null}
    cat <<EOF > patch.yaml
    cluster:
      network:
        cni:
          name: none
    EOF
    ```

    ```bash theme={null}
    talosctl gen config \
        my-cluster https://mycluster.local:6443 \
        --config-patch @patch.yaml
    ```
  </Tab>

  <Tab title="Without kube-proxy">
    If you want to deploy Cilium without kube-proxy, you also need to disable kube proxy:

    ```bash theme={null}
    cat <<EOF > patch.yaml
    cluster:
      network:
        cni:
          name: none
      proxy:
        disabled: true
    EOF
    ```

    ```bash theme={null}
    talosctl gen config \
        my-cluster https://mycluster.local:6443 \
        --config-patch @patch.yaml
    ```
  </Tab>
</Tabs>

### Installation using Omni

If you are using [Omni](https://www.siderolabs.com/platform/saas-for-kubernetes/), you can deploy Cilium using the [manifest sync](../../omni/cluster-management/sync-kubernetes-manifests) feature in a cluster template.

**Step 1.** Create a `values.yaml` file with the Cilium Helm values:

```yaml theme={null}
ipam:
  mode: kubernetes
kubeProxyReplacement: false
securityContext:
  capabilities:
    ciliumAgent:
      - CHOWN
      - KILL
      - NET_ADMIN
      - NET_RAW
      - IPC_LOCK
      - SYS_ADMIN
      - SYS_RESOURCE
      - DAC_OVERRIDE
      - FOWNER
      - SETGID
      - SETUID
    cleanCiliumState:
      - NET_ADMIN
      - SYS_ADMIN
      - SYS_RESOURCE
cgroup:
  autoMount:
    enabled: false
  hostRoot: /sys/fs/cgroup
```

If deploying without kube-proxy, also add:

```yaml theme={null}
kubeProxyReplacement: true
k8sServiceHost: localhost
k8sServicePort: 7445
```

**Step 2.** Render the Cilium manifests using Helm:

```bash theme={null}
helm repo add cilium https://helm.cilium.io/
helm repo update
helm template \
    cilium \
    cilium/cilium \
    --version 1.19.3 \
    --namespace kube-system \
    --values values.yaml > cilium.yaml
```

**Step 3.** Reference the rendered manifest in your Omni cluster template using the `file` field:

<CodeBlock lang="yaml">
  {`kind: Cluster\nname: my-cluster\nkubernetes:\n  version: ${k8s_release}\n  manifests:\n    - name: cilium\n      file: cilium.yaml\n      mode: one-time\npatches:\n  - name: disable-default-cni\n    inline:\n      cluster:\n        network:\n          cni:\n            name: none\n...\n# Include machines for template`}
</CodeBlock>

If deploying without kube-proxy, also disable it in the patches:

```yaml theme={null}
patches:
  - name: disable-default-cni-and-proxy
    inline:
      cluster:
        network:
          cni:
            name: none
        proxy:
          disabled: true
```

**Step 4.** Apply the cluster template:

```bash theme={null}
omnictl cluster template sync --file cluster-template.yaml
```

Omni will wait until the Kubernetes API is available and the cluster is healthy before applying the Cilium manifests. See [Sync Kubernetes Manifests](../../omni/cluster-management/sync-kubernetes-manifests) for more details on manifest sync modes and status monitoring.

### Installation using Cilium CLI

> Note: It is recommended to template the cilium manifest using helm and use it as part of Talos machine config, but if you want to install Cilium using the Cilium CLI, you can follow the steps below.

Install the [Cilium CLI](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/#install-the-cilium-cli) following the steps here.

<Tabs>
  <Tab title="With kube-proxy">
    ```bash theme={null}
    cilium install \
        --set ipam.mode=kubernetes \
        --set kubeProxyReplacement=false \
        --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
        --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
        --set cgroup.autoMount.enabled=false \
        --set cgroup.hostRoot=/sys/fs/cgroup
    ```
  </Tab>

  <Tab title="Without kube-proxy">
    ```bash theme={null}
    cilium install \
        --set ipam.mode=kubernetes \
        --set kubeProxyReplacement=true \
        --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
        --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
        --set cgroup.autoMount.enabled=false \
        --set cgroup.hostRoot=/sys/fs/cgroup \
        --set k8sServiceHost=localhost \
        --set k8sServicePort=7445
    ```
  </Tab>

  <Tab title="Without kube-proxy + Gateway API">
    Install [Gateway API CRDs](https://docs.cilium.io/en/stable/network/servicemesh/gateway-api/gateway-api/#prerequisites), then install Cilium:

    ```bash theme={null}
    cilium install \
        --set ipam.mode=kubernetes \
        --set kubeProxyReplacement=true \
        --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
        --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
        --set cgroup.autoMount.enabled=false \
        --set cgroup.hostRoot=/sys/fs/cgroup \
        --set k8sServiceHost=localhost \
        --set k8sServicePort=7445 \
        --set gatewayAPI.enabled=true \
        --set gatewayAPI.enableAlpn=true \
        --set gatewayAPI.enableAppProtocol=true
    ```

    > Note: If you plan to use gRPC and GRPCRoutes with TLS, you must enable ALPN by setting `gatewayAPI.enableAlpn=true`.
    > Since gRPC relies on HTTP/2, ALPN is required to negotiate HTTP/2 support between the client and server.
  </Tab>
</Tabs>

### Installation using Helm

Refer to [Installing with Helm](https://docs.cilium.io/en/stable/installation/k8s-install-helm/) for more information.

First we'll need to add the helm repo for Cilium.

```bash theme={null}
helm repo add cilium https://helm.cilium.io/
helm repo update
```

<Tabs>
  <Tab title="Helm install">
    After applying the machine config and bootstrapping Talos will appear to hang on phase 18/19 with the message: retrying error: node not ready.
    This happens because nodes in Kubernetes are only marked as ready once the CNI is up.
    As there is no CNI defined, the boot process is pending and will reboot the node to retry after 10 minutes, this is expected behavior.

    During this window you can install Cilium manually by running the following:

    ```bash theme={null}
    helm install \
        cilium \
        cilium/cilium \
        --version 1.18.0 \
        --namespace kube-system \
        --set ipam.mode=kubernetes \
        --set kubeProxyReplacement=false \
        --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
        --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
        --set cgroup.autoMount.enabled=false \
        --set cgroup.hostRoot=/sys/fs/cgroup
    ```

    Or if you want to deploy Cilium without kube-proxy, also set some extra parameters:

    ```bash theme={null}
    helm install \
        cilium \
        cilium/cilium \
        --version 1.18.0 \
        --namespace kube-system \
        --set ipam.mode=kubernetes \
        --set kubeProxyReplacement=true \
        --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
        --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
        --set cgroup.autoMount.enabled=false \
        --set cgroup.hostRoot=/sys/fs/cgroup \
        --set k8sServiceHost=localhost \
        --set k8sServicePort=7445
    ```

    And with GatewayAPI support:

    ```bash theme={null}
    ...
        --set=gatewayAPI.enabled=true \
        --set=gatewayAPI.enableAlpn=true \
        --set=gatewayAPI.enableAppProtocol=true
    ```

    After Cilium is installed the boot process should continue and complete successfully.
  </Tab>

  <Tab title="Helm manifests install">
    Instead of directly installing Cilium you can instead first generate the manifest and then apply it:

    ```bash theme={null}
    helm template \
        cilium \
        cilium/cilium \
        --version 1.18.0 \
        --namespace kube-system \
        --set ipam.mode=kubernetes \
        --set kubeProxyReplacement=false \
        --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
        --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
        --set cgroup.autoMount.enabled=false \
        --set cgroup.hostRoot=/sys/fs/cgroup > cilium.yaml

    kubectl apply -f cilium.yaml
    ```

    Without kube-proxy:

    ```bash theme={null}
    helm template \
        cilium \
        cilium/cilium \
        --version 1.18.0 \
        --namespace kube-system \
        --set ipam.mode=kubernetes \
        --set kubeProxyReplacement=true \
        --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
        --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
        --set cgroup.autoMount.enabled=false \
        --set cgroup.hostRoot=/sys/fs/cgroup \
        --set k8sServiceHost=localhost \
        --set k8sServicePort=7445 > cilium.yaml

    kubectl apply -f cilium.yaml
    ```
  </Tab>

  <Tab title="Helm manifests hosted install">
    After generating `cilium.yaml` using `helm template`, instead of applying this manifest directly during the Talos boot window (before the reboot timeout).
    You can also host this file somewhere and patch the machine config to apply this manifest automatically during bootstrap.
    To do this patch your machine configuration to include this config instead of the above:

    ```bash theme={null}
    cat <<EOF > patch.yaml
    cluster:
      network:
        cni:
          name: custom
          urls:
            - https://server.yourdomain.tld/some/path/cilium.yaml
    EOF
    ```

    ```bash theme={null}
    talosctl gen config \
      my-cluster https://mycluster.local:6443 \
      --config-patch @patch.yaml
    ```

    However, beware of the fact that the helm generated Cilium manifest contains sensitive key material.
    As such you should definitely not host this somewhere publicly accessible.
  </Tab>

  <Tab title="Helm manifests inline install">
    A more secure option would be to include the `helm template` output manifest inside the machine configuration.
    Using the `patch.yaml` created in the [machine configuration preparation](#machine-configuration-preparation) step, patch this into your machine configuration:

    ```yaml theme={null}
    cluster:
      inlineManifests:
        - name: cilium
          contents: |
            --
            # Source: cilium/templates/cilium-agent/serviceaccount.yaml
            apiVersion: v1
            kind: ServiceAccount
            metadata:
              name: "cilium"
              namespace: kube-system
            ---
            # Source: cilium/templates/cilium-operator/serviceaccount.yaml
            apiVersion: v1
            kind: ServiceAccount
            -> Your cilium.yaml file will be pretty long....
    ```

    This will install the Cilium manifests at just the right time during bootstrap.

    Beware though:

    * Changing the namespace when templating with Helm does not generate a manifest containing the yaml to create that namespace.
      As the inline manifest is processed from top to bottom make sure to manually put the namespace yaml at the start of the inline manifest.
    * Only add the Cilium inline manifest to the control plane nodes machine configuration.
    * Make sure all control plane nodes have an identical configuration.
    * If you delete any of the generated resources they will be restored whenever a control plane node reboots.
    * As a safety measure, Talos only creates missing resources from inline manifests, it never deletes or updates anything.
    * If you need to update a manifest make sure to first edit all control plane machine configurations and then run `talosctl upgrade-k8s` as it will take care of updating inline manifests.
  </Tab>
</Tabs>

## Known issues

* There are some gotchas when using Talos and Cilium on the Google cloud platform when using internal load balancers.
  For more details: [GCP ILB support / support scope local routes to be configured](https://github.com/siderolabs/talos/issues/4109)

* When using Talos `forwardKubeDNSToHost=true` option (which is enabled by default) in combination with cilium `bpf.masquerade=true`.
  There is a known issue that causes `CoreDNS` to not work correctly.
  As a workaround, configuring `forwardKubeDNSToHost=false` resolves the issue.
  For more details see [the discussion here](https://github.com/siderolabs/talos/pull/9200)

## Other things to know

* After installing Cilium, `cilium connectivity test` might hang and/or fail with errors similar to

  `Error creating: pods "client-69748f45d8-9b9jg" is forbidden: violates PodSecurity "baseline:latest": non-default capabilities (container "client" must not include "NET_RAW" in securityContext.capabilities.add)`

  This is expected, you can workaround it by adding the `pod-security.kubernetes.io/enforce=privileged` [label on the namespace level](../security/pod-security).

* Talos has full kernel module support for eBPF, See:
  * [Cilium System Requirements](https://docs.cilium.io/en/stable/operations/system_requirements/)
  * [Talos Kernel Config AMD64](https://github.com/siderolabs/pkgs/blob/main/kernel/build/config-amd64)
  * [Talos Kernel Config ARM64](https://github.com/siderolabs/pkgs/blob/main/kernel/build/config-arm64)